Agency TRANSPORTATION DEPT

Contractor Esri

SWIFT Contract Number 55744

Project Name Linear Referencing System (LRS)

Project Number (if applicable) P1423

Summarize purpose of the contract, including why it was necessary: MnDOT sought a fully configured, tested and operational LRS to be Minnesota's authoritative source for roadway location data.

Project Duration

Start Date Thu, 12/20/2012 - 00:00

Original end date Mon, 06/30/2014 - 00:00

Actual end date Tue, 06/30/2015 - 00:00

Did project finish on time? No

Total billable hours utilized (if applicable) {Empty}

Contract Amounts

Original contract: $1364300

Amendments {Empty}

Contract total $1289300

Did project finish within budget? Yes

What was the funding source? TIS Modernization Budget

Is this a sole source contract? Yes

If this was a single source contract, explain why the agency determined there was only a single source for the services (if applicable) Esri’s Roads and Highways software was available to MnDOT as part of the State’s Enterprise License Agreement. This contract was for consulting services to implement the software within the MnDOT environment. Only Esri could do this work since the Roads and Highways product was not available through a reseller nor were other consultants able to implement this software for an organization - only Esri could. The Esri Roads & Highways product met MnDOT's business requirements and would be part of the needs for mainframe retirement. It is a Commercial Off-The-Shelf (COTS) solution which was aligned with MnDOT's system modernization strategies. This purchase was an extension of the enterprise license agreement MnDOT already had with Esri. Costs incurred were for professional/technical services for customization of the software and data conversion.

Evaluator Name Jocelyn Stein

Evaluator Email Address jocelyn.stein@state.mn.us

Evaluate the contractor's timeliness Dissatisfied.

Project Management: Project Plan, status reports, updated timeline and other project management duties were not delivered on a regular basis.

Technical Assistance: Response time was often delayed because correct people were not available on weekly calls. Technical staff were unfamiliar with MnDOT specific details and often answered questions with what will be available in future vs. resolving present issue(s). Believe that if Esri provided on site staff up front to work with DOTs on the products they will be using, how they integrate, what is “really” feasible, and the options for configuration the timeline of projects could be met. Many times conference calls were ineffective and only a portion of the issue was resolved. Many times the customer does not know enough about the product to ask the type of questions they need answered.

Data Conversion: Data conversion and the delivery of “frozen data” was not planned for timely. Team attended Data Migration Overview and Training where decisions were made and documented, however, data conversion took over a year and was still not delivered completely.

Deliverable documents: Deliverable documents were delivered usually within a month of timeline, however, were vague, cookie cutter, not specific to MnDOT’s project, and needed multiple cycles of review/approval. In some cases we needed to remind Esri that we had not approved the documents so they would follow-up.

Testing/Test Scripts: These were delivered late, not related to proper version of software, and final delivery was sent with only days to review before formal UAT acceptance was scheduled.

Data Reviewer: Questions regarding this were delayed and demonstrations were not specific to our questions.

As a result project was delayed multiple times.

Evaluate the contractor's quality Dissatisfied.

Project Management: Project Plan, status reports, updated timeline and other project management duties were often incorrect and/or incomplete. MnDOT staff had to spend a large amount of time reviewing and correcting what was delivered. MnDOT requested (and received) a new/different PM for later parts of the project. This did improve project management practices for the last few months of the project.

Technical Assistance: Esri never gathered specific detailed requirements and therefore, MnDOT spent a lot of time trying to figure out configuration, event behaviors, workflow manager, data reviewer and RCE.
Technical staff were unfamiliar with MnDOT specific details and often answered questions with ideas vs. actual solutions to our issue. Esri often provided an expert for one piece of our solution but did not have an enterprise level expert that would explain how changing one configuration or setup in one area/software tool could or would impact another area or software tool.

Data Conversion: Requirements were not gathered and documented thoroughly prior to conversion being started. Team attended Data Migration Overview and Training where decisions were made and documented, including architectural and performance requirements, however, none of our pilot data was not delivered to those specifications and were told “we thought you’d like to follow what New York did”.
Our state required THREE complete state-wide data loads because of misunderstood requirements, however, were rarely contacted by Esri with questions about MnDOT requirements during the process. Because of delays and inaccurate conversion, MnDOT ended up creating their own conversion processes for over 40 events and tracking each detail of their conversion which cost both time and resources for our department.

Deliverables: Many deliverables contained other states names instead of Minnesota, which led us to believe that the deliverables are just changed per project and not created specifically for the state they are contracted with.
Deliverables were usually vague, and not specific to MnDOT’s project requirements, or architecture.
Deliverables that were planning deliverables such as Testing, were not complete or usable within the timeline of the project.

Performance Testing/Scalability: MnDOT was given recommendations and also provided our architecture so that performance could be tested. Esri did not test in the same environment as MnDOT, and did not test with the same software requirements/architecture they recommended. This deliverable was rejected as it had no value to our current environment or requirements for performance.

Testing/Test Scripts: Only 22 UAT scripts. Many of these tests were just signing into the system, and doing basic functionality not specifically related to MnDOT business processes. They were written on high level requirement, with missing scripts, and some requirements were not delivered. None of the combined process tests used MnDOT business processes of using past dates.

Defects/Bugs: Some of our defects reported were created or caused because of lack of understanding on configuration and how software works. Critical defects were pushed to the next Esri release. During our project, we had changed version from 10.2.1, 10.2.2 and then ultimately deploying with 10.3.1 in order to address some of the major issues with backdating effective dates. This release did not address the issue either.

Training: End user training was marginal. Training did not include complex business process scenarios, just basic sign-on, and performing basic tasks such as create route, realign route, and reassign route.

Data Reviewer: Questions regarding this were delayed and demonstrations were not specific to our questions.

As a result project was delayed multiple times.

Evaluate the contractor's cost Neutral

Evaluate the contractor's overall performance Dissatisfied, by the close of the contract it was clear that Esri did not adequately understand the needs of MnDOT to offer guidance and technical solutions, nor did the Esri team fully understand and/or communicate the limitations of their product. As a result MnDOT delayed the project in order to acquire a later version of the product under the pretense that it would correct a critical software deficiency (back dating). This turned out to be false.

Esri's IT Staff were open to taking support calls and helping us work through the issues even after the contract had closed. Although it does not change the fact that the deliverables were late, incomplete or simply wrong, MnDOT appreciates that they were willing to help us work through the issues that got created.

Would you engage the contractor's services again? Yes

Describe any negative action taken during the duration of this contract (i.e., termination, pursuit of suspension, etc.): Some deliverables rejected or changed. Requested (and received) additional or new Project Management staff for the last few months of the project. MnDOT negotiated an approximately $35,000 discount on rejected deliverables. In hindsight perhaps other action should have been taken re: conversion delays.

Please rate the overall experience with the contractor 2 - dissatisfied