It's amazing how often a problem that seems unsolvable while you're working on it ends up having an easy solution after you put it aside for a while.
Several months ago I wrote about a problem with a Measurement Specialties barometric pressure sensor. I had come to the conclusion that either the sensor was faulty, or I had damaged it during installation on the board. I kept putting off desoldering it, partly because it's a challenging part to solder, and partly because I only had two spares, and they're expensive; I didn't want it to be faulty.
Well, late last night I got the bug to look at it again. The data sheet shows the calculations that need to be made to get a calibrated result, and shows "typical values" for each of the six factory calibration parameters, uncalibrated pressure and temperature measurements, and each step of the process. It never dawned on me that those values might all be part of one measurement and calculation operation.
So I wrote a small app on the Mac that used the same calculation code that was on the sensor board, but put in the example values instead. Sure enough, the result I got did not match, and I started looking into the intermediate results. I noticed one of those was exactly double the example value, and that got me looking at the implementation of the equation. Looking very closely at the data sheet, I started re-writing the equations. Turns out, the code I had found on their site was incorrect, and the code I wrote based solely on the data sheet worked correctly.
For reference, here is C/C++ code that works. mC1 through mC6 are the calibration parameters from the device ROM. mRawTemperature and mRawPressure are the raw sensor readings. mTemperature and mPressure are the final, calibrated result. Temperature is in degrees Celcius * 100, so you have to divide the result by 100 to get the temperature. Pressure is in millibars * 100, so do a similar division to get mb.
[cpp]
{
…
int64_t dT = mRawTemperature - mC5 * 256LLU;
mTemperature = 2000LLU + dT * mC6 / 8388608LLU;
int64_t offset = mC2 * 65536LLU + dT * mC4 / 128LLU;
int64_t sens = mC1 * 32768LLU + dT * mC3 / 256LLU;
mPressure = (mRawPressure * sens / 2097152LLU - offset) / 32768LLU;
…
[/cpp]
This sensor should prove to be very accurate, and will give us the balloon's pressure altitude, as well as the temperature inside the insulated payload box. It's only real drawback is a lower pressure limit of 10 mbar, corresponding to an altitude of about 26 km (~85 kft). We're hoping to go past 30 km (~100 kft). Hopefully the GPS will be a good backup.