ASSOCIATES (vol. 2, no. 2, November 1995) - associates.ucr.edu
EVALUATING REPSONSES TO THE TREASURY'S REQUEST FOR PROPOSAL or Selection by Numbers by Rosalind Coote Project Team Member/DBA Treasury Information Centre The Treasury Wellington, New Zealand Background In a previous article for _ASSOCIATES_, John Lozowsky described the creation process for the RFP document. In the spirit of continuing the story, I will take up the part where we selected the product and supplier we would pilot. Rest assured the story does have a happy ending; as I write this we have 163 real users trained on the database we finally set up! I should have run a mile when I got a folder two inches thick given to me on December 24th 1993 and told to read it through over the holiday! This tome was the supplier responses to the Treasury's Request for Proposal (RFP). All members of the evaluation team met again in January 1994. The people involved were: three members of the Information Services team including myself; the network administrator; and three representatives of the user community, Mary Anne, Peter and Jonathan. Remember those names - they are important! In terms of technical and user viewpoints we had a even split really because at that time I had little technical knowledge but had used a lot of databases. My role until then had been liaising with users to prioritise their needs for the new system. We were evaluating the five tenders that made it through the first cut. Information Management had already rejected two for not complying with the platform (Unix based solutions were out), or clearly inadequate security (being a government agency with responsibility for economic advice this is a top consideration). The first step for the evaluation team was to establish how we wanted to weight the responses. This is like saying - in which areas is the right answer worth more? After a lot of discussion we came up with: 50% for Functionality (how many things we required could be provided) 20% for Support (how well could the supplier support the product) 20% for Vendor (questions asked to verify quality of vendor) 10% for Technical (how well did the tender meet technical requirements) The Technical weighting seems low - but took into account that this covered things like backups and drivers, not what the product actually supplied. The Support and Vendor seems high, but for Treasury we viewed it as a high risk area. Being based in New Zealand isolation from the vendor causes real problems which we had encountered the sharp end of with our previous database. Our site does a lot of customization work on anything we put in, so we were also looking for a company with strong client focus and preferably New Zealand based. For each question on the database tender the group agreed on a mark for each supplier. There were 166 questions times 4 vendors to get through. Each question could get a mark of 10, 5 or 0. 10 = product fully meets our requirement 5 = product meets our requirement half-way 0 = product cant meet requirement at all Suppliers write tenders to make it read like their product can do anything and everything so we looked at answers with a sceptical eye. The other thing we did was to adjust marks according to the priority users gave them. The priority was given a weighting out of ten. Priority as percentage of mark = final mark 10 = Mandatory 6 = Highly desirable 5 = Desirable 2 = Nice to have For example : Requirement. The system should be able to track updates and amendments to documents. (Desirable) A mark of 10 is downgraded to 5 because the requirement was rated as desirable although it was fully met by the tender. There are a lot of different ways to do the numbers when looking a tender. This is just one way, chosen with the help of evaluation team members Mary-Anne, Peter and Jonathan. They kept the quality of our process very high indeed. They were aiming for a method that would stand up to audit analysis (should it transpire!) and applied the techniques of their job (economic analyst) to ensure a sound process was used. I'd recommend getting number-crunchers types on your evaluation team! One of the questions (Vendor quality) wasn't marked in quite such a rigorous way. As a team we decided this was a judgment call type question and we just gave the answers an overall mark eg. how long has the supplier been in business, how many qualified staff employed. End result of the evaluation was (1) a spreadsheet documenting how each product stacked up and (2) a team decision that only two products were worth seeing demonstrated. Demo Time!! The two suppliers had about a month to get a demonstration together. The reason we gave them this long is because (1) we wanted them to use sample data from our current system and therefore they needed to convert it and (2) we were telling them the sequence of what we wanted to see in the demo. The demo script I wrote was the same for each supplier. One supplier (Supplier B) took this very seriously indeed and hired in a overseas consultant just to set up the demo database and give the demonstration. This person visited me, looked at the current system and got an idea of what we were aiming for (we thought that was quite ethical as the other supplier - Supplier A - had worked with our database for 5 years already). Needless to say, Supplier B's demo was the best by far. The demos were two hours long. The first hour was to show how the users would access the system. The users at the demo (Jonathan, Mary Anne and Peter) would leave after the first hour. Second hour was for the technical staff to see the DB administration functions and ask technical questions. The script we supplied was useful for checking our needs against - eg. if they glossed over printing we could say "show us how you print the hit list again". Having spent about 6 months working up to seeing a product demonstration we were amazed to find a product that impressed us mainly because of it's simplicity and "up-market" interface. The users knew straight away that this was the product they wanted us to pilot and we were happy to agree because of the evaluation process results and demonstration they had been able to provide. The we checked our scores again... A useful exercise we did was to go back and adjust our scores (up and down!) after seeing the demos. This wound up the evaluation process nicely. Reference Checks As a final recommendation, reference checks can provide some fascinating information. We rang sites in the States and UK that had been set up for a while and got some useful information about what it was like to put in the product. If there are some competing suppliers for the product this is a good time to find out how good they are and what their support might be like ! Keeps your supplier on their toes... Summary: . have some users as members of the evaluation team . agree on weighting for different sections of the tender . you can also weight answers by how users prioritised it . send the supplier a demonstration script that they have to follow . reconfirm scores after seeing the demos . check references by asking other sites some set questions