Converting Data to HIFIS 4: What You Should Know

Most communities considering using HIFIS 4 are not starting from scratch. The truth is, in a pre-HIFIS 4 world, communities did something to get data for reporting, and in a lot of cases, they don't want to lose all of that prior data.

This situation leads many communities to consider what to do with their previous data, when they switch to HIFIS 4. If you're having these conversations, you're not alone! There are lots of other communities struggling with the same question.

What are the Options?

The first question to ask is "what are the options?" You can't make an informed decision about how to proceed without knowing what your choices are.

There are three main ways you can proceed, depending on your current data system:

1. If currently using HIFIS 3, there is a HIFIS 3 to 4 conversion tool made available by the HIFIS Development Team;
2. If currently using a different database, you could pay a third-party software developer to build a conversion tool for you; or,
3. You could leave your data in its current data system and start HIFIS 4 with a blank slate.

The second option, paying a third-party software developer, is likely to be very expensive and is not recommended for most communities. Therefore, most communities will be choosing between converting from HIFIS 3 to 4, and not converting.

Converting Data: General Considerations

Consent

Most legislation surrounding the collection, use, and disclosure of personal information states that when a client consents to have their information collected, it can be used and disclosed only for specific purposes. When converting from a non-shared data system to a shared data system, most communities therefore need to develop and implement new consent forms that include a clause stating that the client's information can be shared with other agencies.

Let's assume, then, that your community is developing a new consent form for use with HIFIS 4.

If you have a new consent form that states that information can be shared, and you begin using it as you move to HIFIS 4, that means all your previous clients have been signing consent forms that do not allow their data to be shared.

Let's further assume that you've been using your previous data system for some period of time, like 5 years. In that time, you've worked with a number of clients - hundreds, or even thousands - who have since been housed and are no longer interacting with your homeless-serving system. Or who have moved on to another city. Or who are currently spending time in an institution.

It is nearly impossible for your community to ensure that you have valid consent to share personal information for all your past clients, if you weren't initially collecting consent to do so from the start. What this means is you're going to have some sub-section of clients in your current database whose data cannot be converted while still satisfying the terms of the consent they've provided.

Similarly, you may have previously been collecting consent that was valid for a period of time, like one year, and could have a large number of past clients who have expired consent. So for them, you have no permission to do anything at all with their data.

If you want to proceed with data conversion, you'll need a way to ensure that you're adhering to the legislation that allows you to collect personal information. This may mean converting only a select group of clients (those with appropriate consent), or it may mean converting all data but blocking access to the data for those who did not previously have permission to access it.

It's strongly recommended you consult with a legal adviser if you're considering converting your previous data.

Data Clean-Up

A second topic to consider is the issue of data clean-up. At its simplest form, you may be considering taking one database and converting it from one format to another. This alone carries some risks of certain fields not transferring properly or data getting corrupted in the transfer.

You are going to need to do at least one test conversion, and then perform a number of tests to see whether the conversion worked properly or not, identify what data pieces did not convert adequately, and address those issues. This is an iterative process, and there's no telling how long this step will take. Resources - in particular IT support - will need to be assigned to this task.

The problem is further compounded if you are considering taking two or more existing databases and merging them into one HIFIS 4 database. This is a likely scenario, considering that many communities might have two or more shelters, and the most common arrangement is for each shelter to be using an isolated HIFIS 3 installation.

If you're considering merging multiple databases, you're going to have the added challenge of addressing duplicate clients. Even at its very best, software (in general) can only successfully identify duplicate people some of the time. Unfortunately, HIFIS 4 is not the best software at identifying and addressing duplicate records.

There are several challenges with locating duplicate records. You might collect minimal information on clients and only have their name and age, or even approximate age.  With limited information on each client, it's hard to tell whether there's an actual duplicate or two distinct clients with the same name. There could easily be two 40-year-old James Smiths or two 30-year-old Emily Martins. In fact, among my own group of acquaintances I know two different Daniel MacDonalds who live in the same city, have the same hobbies, and are about the same age!

Even if you collect more than just the bare minimum, there's still a challenge.  For example, two clients with a date of birth of 1980-03-06 and 1980-06-03 might look to a computer like two different clients, but one of the files might have been a data entry error from a staff who doesn't remember if date of birth should be entered as YYYY-DD-MM or YYYY-MM-DD.

Then there's spelling errors and nicknames. For example, is William Jonson the same as Will Johnson? What about David Thomas-Brown and Dave Brown (and David Thomas)? Then you also could very well have clients who have changed their names after a marriage, divorce, or gender transition.

The point is, after conversion your merged database is going to be messy.  You can manually merge duplicate clients in HIFIS 4, but you can only do so one at a time, so it can take a while if you have a large database. Several communities that have successfully merged databases have done so with the assistance of many dedicated resources, such as assigning extra personnel to data clean-up after merge.

It's recommended that if you're considering converting your database, you have a frank conversation with your project team and your IT support and determine how many resources are available to this step. If you find yourself at or over your budgeted resources, it may be a good idea to abort the idea of data conversion and start HIFIS 4 from a clean, blank database.

What to Convert

So far, we've been assuming that you want to convert your client files, but the question remains: how much of the content within them do you want to/need to convert? You probably want to convert shelter stays, but what else? Do you need all the case notes, all the incident reports, and all the past phone numbers?

Back to consent for a moment - it's possible that you were previously collecting some information (like health-related information) that your legal advisers are now telling you you cannot be collecting and storing in a shared database. So there might be some pieces of information that you're not allowed to convert, even if the rest of the conversion project has been green-lighted by legal.

You'll therefore need to decide what pieces of data you'll be converting and what you'll decide to leave behind. If you're hiring a third-party software developer to build a conversion tool for you, this is an important decision. In general, the more information you want to convert, the more complicated the process and the longer it will take and the more it will cost. Even without the factor of a third-party developer, more data being converted means an increased risk that something will not transfer properly.

If embarking on a data conversion project, it's recommended that you consider carefully what your data requirements are before you proceed.  It's easy enough to say that you'll just convert the whole database, but you may be inadvertently costing resources in order to convert a piece of data that's ultimately not going to be important to anyone.

The HIFIS 3-4 Conversion Tool

There is a tool put out by the HIFIS Development Team that allows you to convert your HIFIS 3 database to HIFIS 4. However, it does have some idiosyncrasies of its own.

The first thing to understand is it's not very commonly used. While a number of communities are using HIFIS 3 today, and a number of communities are using HIFIS 4 today, a very small number of communities (perhaps 0) are using the conversion tool today. As a result, more software development resources are allocated to fixing bugs and adding new features to HIFIS 3 and 4 than to the conversion tool. If you call the support desk today and ask for the conversion tool, they may need to update the conversion tool (since it hasn't been updated in 6 months, and there are new features in HIFIS that it doesn't accommodate) before you can use it. There also may be bugs in the software, or it may cause bugs in your HIFIS 4 database that wouldn't be present if you were starting with a clean, blank database.

Related, you're going to need to work with the HIFIS Development Team to do the conversion. It's quite likely you'll try a test conversion and something won't quite work as intended. You can expect to spend some time talking to the support desk and working through these issues, getting a few different updates and iterations of the conversion tool, and trying again.  (Note that you can also expect the same thing if you have a custom conversion tool developed by a third party.)

Third, it's going to convert all of your clients, including the past ones. Their consent status should transfer over (see A Primer on Consent in HIFIS 4) but for better or worse, you'll be converting your entire past history of clients, including ones you haven't seen in ten years. This could potentially make your database very large, and if you're server's not built for that amount of data, slow down the software significantly.

Finally, you have limited choice about what data gets converted and what doesn't.  You can select some options in the tool, but there are some pieces you're stuck with. For example, it'll convert all your Housing Units (which some communities have stated results in a lot of duplication) and all your People (which could be cause for some concerns regarding privacy).

Not Converting

If you've gotten to this part in this article, you may be feeling concerned about the idea of converting your previous database. To be honest, you should be. There are a lot of considerations to take into account, and even if you decide you can legally and ethically go ahead with the conversion, it's going to take a lot of resources to do so.

Perhaps, then, you might be thinking that not converting data is the right approach. It's certainly simpler to manage. Also, remember that we aren't suggesting destroying all your past data. If you really need to access some data that did not get converted, you can go and check your old database.

But now we must consider the impact of not converting data, in particular on our front-line staff at our service providers.

If data is not converted, then each current client will need to be manually re-entered into HIFIS 4. The default way this occurs is the next time a client interacts with a service provider, staff will get them to sign a new consent form, and will open a new file in HIFIS 4. However, this could considerably slow down the workflow at service providers, particularly busy shelters.

One way to mitigate this problem is to begin collecting consent to share data well before (i.e. several months before) HIFIS 4 launch date. Then, at least front line staff don't need to go through the hassle of both obtaining new consent AND using a new database all in the same day.  Theoretically, on launch day, the majority of clients seen that day have already signed a new consent form, which would save some time. Instead of having to obtain consent, create a new file, and book a client in, they would just have to create a new file and book the client in.

A further step that can be taken is to allocate some resources to data entry. For example, perhaps the municipality can hire a student or temp to create client files, prior to launch day, after clients have signed the new consent form. This would further streamline launch day for front line staff. Instead of having to create a new file then book the client in, they would just have to book the client in. However, there's a balancing act here between how much resources to be spent in advance manually entering data, and how much front line staff can do going forward.

Even if you take these steps, front line staff may find it annoying that all their case notes are entered into one data system, and now they must continue case management in a new system even though they've been working with the client for months or years, but it's not overly onerous to copy and paste text from one source to another, especially considering this will only be the case for a finite number of cases.

Conclusion

Change is difficult. It's pretty easy to keep doing the same thing, even when it's not the best thing. So if you decide to switch to HIFIS 4, there will be bumps in the road, and data conversion is one of the major speed bumps. Whether you decide to convert data or not will be a decision you make after considering your resources, your data needs, and what's important in your community.