HAB1 04.27.2016 Ascent to 31K Meters (103,000 Feet)

I am Kevin Hubbard of Black Mesa Labs. I am a High Altitude Space Balloon Engineer and this is my story of the last 3 days:

I work with a small group of ultrasound engineers in Issaquah,WA-USA who are also the Balloongineers – 1 of 7 teams in Washington State that participate in the Global Space Balloon Challenge. There are 398 teams across the planet that participate each year in the challenge.

Last month I was invited to join the local Issaquah team. Someone from my past must have told the Balloongineers that I worked on the fault tolerant AS/400 at IBM in the early 90’s ( descendant of the Space Shuttle’s AP-101 Flight Computer ). I certainly wasn’t recruited into the Balloongineers for any high altitude bravery. As a quick introduction to the team as the greenhorn they asked me to use all of my Black Mesa Labs’ Maker resources to design and build a new electronics module for Flight Navigation, Data recording and live 2-way Satellite Communications to earth – all in 6 weeks. Go Big or Go Home – sounds fun!

If you would prefer to skip the story, there are YouTube videos at the end of the story. Technical details for HAB1 including block diagram and descriptions are at the very end of this blog. Grab a cup of coffee.

— Story Begins —

HAB1_04_27_2016_Electronics.png

I spent 6 challenging weekends non stop doing Hardware and Software development and Test to make the launch date. HAB1 electronics are built from a high altitude flight ready uBlox GPS unit, a 2-way Iridium RockBLOCK satellite communication modem, 1/2 a dozen custom circuits boards design and built using CopperConnection and OSH-Park, a fully custom FPGA “Lizard Brain” design and a RaspberryPi Linux computer running a custom Flight and Communication program written in Python. All these pieces had to work together – all the time – with built in fault tolerance – all in 6 weeks for the contest opening.

IMG_1575.JPG

Sunday : Final hardware assembly 04.24.2016. The 3 10Ah batteries are 132g or about 5 oz each and make up about 1/2 of the weight of the electronics package. The entire electronics package including the batteries weighs in at 870g ( 31oz or almost 2 lbs ). This does not include the styrofoam sphere, 3 solar panels and 3 LEDs. Not bad for a 32bit Linux computer, 30Ah of batteries, GPS, 2-Way SatCom and a custom FPGA board.

Rain tests with the RockBLOCK 2-way Iridium SatCom modem in my driveway did NOT look promising. Driving tests on Sunday even worse. Monday took a long route into work with my convertible with the top down and only 1 in 4 messages got through driving around at  35 MPH ( 60 km/h )  – but the launch window for the contest was this week. I am confident SatCom will work much better when high above the clouds. RockBLOCK can fail or work intermittently in flight I told myself – only the uBlox GPS unit has to work perfectly to feed altitude information to the Flight Computer for fully autonomous operations of flight control for helium venting and ballast pumping. So long as the Flight Computer has reliable flight data, downlink telemetry to Earth can be flaky. OneBusAway has to work reliably all the time – balloon telemetry? balloons are slow – who cares if we miss a few? This isn’t a rocket after all – so I thought.

The Balloongineers Team Leader Jon picks a launch date based on weather predictions for Wednesday – We Go! Final HAB1 pieces all brought together per block diagram.

hab1_mechanical.png

Wednesday : Launch Day 04.27.2016 – Launch site – south end of Lake Sammamish,WA. A large crowd formed on the grassy lawn while the low pressure gas specialists begin filling the HAB1 balloon from a 50L tank containing helium at 2,200 psi or 7,500L ( 2200psi * 50L / 14.7psi = 7483 liters of helium ). The helium begins the slow transfer and expansion to partially inflate the 2000g Kaymont balloon to 2m diameter at 15 psi. I connect power to the Lizard Brain and then boot the Flight Computer. It takes about 10 minutes to boot the Flight Computer ( Linux bootup followed by GPS lock time and then finally POST ) – so I need to get all of my systems running and checked out in advance of launch. Once GPS is locked, the Flight Computer finishes Power On Self Test by exercising the helium vent valve for 3 seconds and then the ballast pump for 2 seconds. Payload estimate for the 3 vessels was 9 lbs, which required 6123 liters of helium based on remaining 400 psi in the tank ( 400psi * 50L / 14.7psi = 1360 liters remaining ).

IMG_1688.JPG

Ten minutes later as expected, POST has finished with the comical site of 2 seconds of ballast liquid squirting out a straw onto the ground. Flight Computer is now a Go for launch.

IMG_1701.JPG

IMG_1701zoom.JPGLow pressure gas expert and valve designer Dr.P and I are now both waiting for the now fully inflated balloon to attach to the valve assembly ( which is connected to the Flight Computer capsule via a 3 wire servo connection ). As the 4″ hose clamp is tightened by the fuel specialists and balloon handlers  – we gasp in horror ( caught on video verbally and by the look of terror on my face here ) as we realize the small 3 pin 0.100″ pitch SIP connector connecting the Flight Computer to the Valve servo assembly is not only disconnected – but the pins are bent. SIP connectors are great – I use them all the time – but they really don’t like to be bent. These were dangerously close to the hose clamp. Bend SIPs once – maybe, bend them twice – and they break off. Do we abort the launch? Is there time to solder a new SIP connector on the leads running 12″ to the flight computer? No – the balloon is filled and ready to go. The balloon handlers in their special balloon protecting gloves are now responsible for keeping the balloon untouched from any foreign objects before flight – it just is not feasible to abort this late in launch with the balloon already inflated.

IMG_1593.JPG

We must fix what we have.  I grab a pair of pliers ( plumber’s pliers mind you – not precision needle nose 22 gauge wire pliers the Flight Computer was built with ) that are on the table for hose clamp assembly. I carefully bend the pins back. Dr.P in charge of the Valve Assembly inspects my work – I plug it into the Valve assembly. The connector is not polarized ( it can be connected backwards – and definitely not work ) and the hose clamp is covering the pin labeling. Is the polarity correct ? Yes. Will it hold? Duct tape it – it must hold. Can we retest the valve preflight? No time, that would either take a full power cycle of the Flight Computer to re-run POST ( 1o minutes ) – or a sprint to ground control ( my laptop ) to type a SatCom message for manual venting. Manual SatCom commands take anywhere from 15-30 minutes to schedule and be applied depending on the current SatCom window. Iridium packets aren’t fast and aren’t inexpensive. Neither retest method are viable. Visual inspection only it has to be – it looks good. Okay – Flight Computer and Valve Assembly are both Go for launch. Balloongineers  we are a go for Launch!

The Balloongineers launched at 2:30PM PST, I mean 21:30 Zulu – I am still the team greenhorn. HAB1 launches with my flight computer capsule, a video+SPOT capsule and finally the ballast vessel.

IMG_1600.JPG

Our backup SPOT GPS ground recovery system failed 5 minutes into flight and immediately we became completely dependent for recovery on the primary Iridium 2-way RockBLOCK SatCom modem and my Flight Computer software and custom hardware design. This was all my new custom design – it ALL had to work now – pressure was on – but I was confident in my work and really was not that concerned. Maybe we should use APRS as the backup system next time – whatever. Primary system is functioning perfectly now and it will work on the ground for recovery. Live detailed telemetry is coming in at anticipated 10-15 minute intervals. Voltage, Current, Temperature, Ascent Rate all Green across the board.

Autonomous picture taken at 109 meters ( 360 feet ) above Sammamish State Park

IMG_0001.jpg

Autonomous picture taken at 455 meters ( 1,500 feet ) above the Issaquah Highlands

IMG_0002.jpg

Autonomous picture taken at 1120 meters ( 3,700 feet ) going north up Lake Sammamish

IMG_0003.jpg

One hour into flight SatCom is still 5×5, reporting in every 10-15 minutes with detailed live telemetry data as scheduled. We were ascending at about 500 ft/min as planned and reached 60K feet per FAA requirements to not hang out in commercial air space for too long. Ascent rate a little high, but manageable. Major ascent correction was not scheduled until 75K feet anyways. Traveling North – ever so slowly.

Autonomous picture taken at 9,772 meters (32K feet ). Note that the Flight Computer inserts the top line telemetry overlay text to each JPEG image while in flight. The CPU looks up nearest city from live GPS coordinates using an on board database of all cities in North America. One of many benefits of using Linux+Python. Numbers on the right are system temperatures, voltages and currents. -3C outside, +25C inside at the batteries. Solar panels generating 228mA at 4.5V. Systems all look great. Insulation is working.

IMG_0021.jpg

Around 00:00 Zulu a regular telemetry downlink message came in – at 80K feet now, but ascent rate of 600 ft/min. Too fast. Something isn’t right, Flight Computer should have autonomously adjusted by venting excess helium in 5 second bursts to slow ascent from 500 ft/min to only 250 ft/min by now. It should have reported this venting back to ground control – it didn’t. Critical scheduled GPS location messages aren’t all getting through though. At 30K feet there was a 25 minute gap between RockBLOCK messages. By (my) design the low priority “! V” venting messages – indicating short 5 second helium vents are posted writes ( fire and forget ) – and are not retried like recovery critical GPS location messages. It is conceivable that the lack of any vent messages is just due to the unreliable SatCom issues. All of a sudden – these low level diagnostic “! V” messages became extremely important. Other than at poweron POST – we haven’t seen any.  Was the electrical SIP connector connection to the valve broken pre-launch? Was the Flight Computer malfunctioning? Altitude is still okay though. I am not fully alarmed yet. Waiting for next anticipated message around 00:10 to 00:15 Zulu to determine if venting is working or not. The next message doesn’t come. No “! V” – no GPS location. Silence. Regularly scheduled SatCom 10 minute telemetry update came and went. Dudes – something isn’t right ( this isn’t NASA after all ) – come here quick. HAB1 has not reported in.

Autonomous picture taken at 35,832 meters (85K feet )

IMG_0043.jpg

Core group of Balloongineers gathers around ground control ( my laptop ). We quickly do some math. Last report of 80K feet and +600 ft/min means 100K ( anticipated balloon burst altitude ) in just 30 minutes from when the last message was received. Mission plan was to hover around 90K feet and hope for some favorable winds to take us east ( there weren’t any ). Uplink command reception and processing from Earth to Space are slow ( 15-30 minutes ) as the HAB has to check for them, and its SatCom window is 10-15 minute when all systems are working flawlessly. Now they obviously were not. Our window to act manually is closing and closing fast. I am starting to question my SIP connection repair to the valve. Maybe the pin broke in flight with the extreme temperature drop to -50C. Core group of us take a vote and all agree – attempt manual override as we have no indication the autonomous flight computer has vented. How much? Maximum. I quickly type “open 45” on my keyboard and double check my syntax and spelling ( yes – even for open ) then click <SEND>. This queues up a manual command via the Iridium Satellite network to override the autonomous Flight Computer and vent excess helium for 45 seconds. 45 seconds – a big vent compared to planned short 5sec burst vents by the autonomous Flight Computer but we need to do this NOW! It is done – now we can only wait. There is no immediate ACK. The message will now sit in an Iridium mailbox system until the HAB1 checks in at regular SatCom intervals and asks if it has any messages for it. These regular SatCom intervals have been quite irregular however – oh and HAB1 by design only checks for uplinks every OTHER downlink message as every uplink check costs an Iridium credit whether you have a message or not. I was trying to be a good steward of Jon’s Iridium credits. I am now regretting that design decision.

Autonomous picture taken at 26,413 meters (87K feet )IMG_0044.jpg

We know now the manual intervention to command a helium vent for 45 seconds was in fact received by HAB1, but unfortunately came in a bit too little, too late as it turns out. Ascent rate was slowed – but not enough, and the HAB kept rising. Regularly scheduled downlink telemetry messages were not coming in. At 5:30pm The Flight Computer issued one last disturbing downlink message home while in flight to ground control on Earth saying “My batteries are fully charged, my electronics are at +20C, but I Have Just FAULTED and REBOOTED myself, I am here ( precise GPS coordinates for a 100×100 foot spot in near space 100,000 feet above Mill Creek ).”. Not good. Not good at all.

Autonomous picture taken at 30,835 meters (101K feet )

IMG_0046.jpg

This was the last the Balloongineers heard from our little High Altitude Balloon in flight. Not a very promising message at that. All was quiet. Semi regular 10 minute telemetry downlink via Iridium was severed. Ground control was completely in the dark. Where is HAB1?

Wednesday night, I slept very little. I checked for new messages from above – but received none. I scoured through the 12 short 50 character messages received during the 3 hour flight but saw no answers to all of my questions. The “! BOOT action_satcom()” message told me a reboot had occurred and that the software was running action_satcom() at the time of the reboot. To have the Lizard Brain force a watchdog reset meant the CPU must have been stuck inside action_satcom() for almost 10 minutes. This function call should take 3 minutes at the most. How? What? Why?

 2:38PM N47.5577 W122.0484 M18,+0 C+22,+27,+28,+26 ! BOOT 0 0
 2:45PM N47.5543 W122.0466 M455,+0 C+20,+28,+29,+27
 2:49PM N47.5497 W122.0448 M1120,+125 Issaquah WA
 2:56PM N47.5485 W122.0459 M1809,+120 C+16,+27,+29,+26
 2:59PM N47.5518 W122.0494 M2473,+120 Issaquah WA 
 3:07PM N47.5617 W122.0582 M3155,+139 V3.9,204,4.5,148
 3:10PM N47.5722 W122.0753 M4136,+139 West Lake Sammamish
 3:14PM N47.5808 W122.0846 M4690,+130 C+16,+28,+29,+26
 3:22PM N47.6053 W122.1088 M5895,+158 C+14,+31,+33,+30
 3:47PM N47.7463 W122.2280 M9772,+186 Inglewood-Finn Hill
 4:03PM N47.8086 W122.2639 M13060,+163 C-4,+25,+23,+19
 4:14PM N47.8092 W122.2390 M14700,+165 Kenmore WA
 4:20PM N47.8045 W122.2346 M15315,+165 C-7,+23,+21,+16
 4:23PM N47.8126 W122.2186 M16322,+189 Bothell WA
 4:33PM N47.8100 W122.1974 M18034,+163 V3.9,263,4.5,227
 4:49PM N47.8120 W122.1724 M20299,+164 V3.9,214,4.4,83
 4:52PM N47.8116 W122.1660 M21378,+212 Maltby WA
 4:56PM N47.8140 W122.1678 M22010,+180 V3.9,259,4.5,197
 5:03PM N47.8169 W122.1698 M23323,+198 V3.9,263,4.5,229
 5:08PM N47.8129 W122.1788 M24585,+186 V3.9,266,4.5,203
 5:35PM N47.8225 W122.1751 M29772,+0 C+12,+22,+23,+15 ! BOOT action_satcom() 9878, 10790

Analyzing the SatCom telemetry data – all systems were green – Voltage, Current, Temperature – all better than could be expected for the harsh near space environment at the time of reboot. My worst case planned for scenario was that a sustained -50C outside environment could bring the Lithium Ion batteries down to -20C and shut down all systems. The FPGA Lizard Brain was designed to prevent this. It continuously monitors battery voltage, current and temperature and has a means to warm the batteries via power resistors in between the cells. The last report were batteries were a warm +22C and fully charged at 3.9V with a predicted current draw of 200mA. Power definitely was not it. Single Event Upset due to excessive alpha ray particles, I planned for that too – everything but GPS and RockBLOCK was wrapped in thick aluminium foil for maximum heat retention, radiation hardening and RF shielding. What happened? Why did HAB1 go silent?

Thursday : I drove in to work and told myself – the balloon must have popped above 100K, the parachute deployed ( as designed ) – it is safely on the ground somewhere around Mill Creek ( or maybe …. Puget Sound ? ). HAB1 will have recognized descent and autonomously exited flight mode and entered recovery mode based on altitude. Recovery mode has a single mission – preserve battery power and phone home – Winter is Coming. The FPGA Lizard Brain will boot the CPU once an hour and phone home with GPS coordinates and then immediately go back to sleep for an hour. This process should start with each day’s rising sun hitting the solar panels. Autonomously ( I modeled my design after the Mars Sojourner rover ). But it didn’t. I myself  was very quiet on Thursday. HAB1 was well, completely silent. Not a peep – all day long. No hourly phone home message – nada. Puget Sound I’m thinking – I bet that is far worse for the RockBLOCK than a rainy drive in my car.

Friday : I drove in to work feeling down – really down. Our balloon was down and we didn’t know where other than it most likely dropped via parachute from 100K – 120K feet above Mill Creek. Big search area. If we were NASA we would simply ask the NSA to retask one of their birds in the sky. We were neither NASA nor NSA however. The balloon itself was one time use expendable, the 2-way RockBLOCK satellite modem, not exactly a one time use disposable item. I arrived at work and started writing a short email as a small gofundme like fund raiser for the purchase of a replacement Iridium RockBLOCK modem to give us one more shot at this year’s competition. The RockBLOCK is a pricey little piece of hardware and I was not okay with Jon buying a 2nd unit himself just because our 1st HAB was lost in space ( or Earth as it may be ). Sparkfun has them in stock ( I just checked ) and amortizing a replacement modem across 10-20 space enthusiastic engineers – all children during NASA’s 1960-1970s Mercury , Gemini and Apollo glory days seemed a logical, moral option. I also could not shake this ridiculous notion to send a HAB2 up to try and locate HAB1. I clicked send – and waited.

Literally minutes later I received an email message – at 8:13AM- not from a generous donor however – but from our humble little HAB1 space Balloon reporting “It is 8:13AM – I am here in Mill Creek,WA at GPS …, … at 300 feet elevation. Please come get me. It is +12C and I am cold” (okay – I embellished a bit on that last part, it did report the +12C, battery status ( still at 80% after 3 days ) and precise GPS location and time – honest ).

N47.8987
W122.1760
M107,+0
C+12,+14,+14,+14
! BOOT
action_shutdown()
63
100693
! V 3
! B 2
3.8 V

I just about fainted. I definitely yelled out ( my neighbors are used to it by now ). I called Terry and Jon the team principals and we jumped in Terry’s truck, made a quick stop at his house for mountain climbing gear( for Mill Creek,WA ? my sister lives there – there aren’t any mountains ) and lots of long poles ( who has this many long poles? ).

IMG_3830.JPG

We headed North during rush hour to a precise location on the ground ( or so I thought ). Now was definitely not the time to complain or question the ridiculous I-405 toll lane tolls.

sheep-on-the-hillside.jpg

107 Meters – I’m picturing in my head our bird on the ground on a nice grassy 300 foot hill surely – nope – not even – after a reasonable in country boots on the ground search area ( Terry also brought 3 pairs of rubber galoshes ) we located our little balloon on top of a tree. A REALLY TALL tree. This picture shows the red capsule with the GoPro and DashCam ( center ) and about 1/2 the tree. Tree grew out of the bottom of a ravine. No snakes at the bottom – but it sure wouldn’t surprise me at this point.

Imported Photos 00001.JPG

So while Terry goes completely Rambo down one side of the ravine, Jon and I go door to door on the other side meeting really nice neighbors of Mill Creek. Jon knocks on doors and explains our situation ( Jon is REALLY good at this by now  – 3rd year at it) and politely asks for permission to enter and walk through their back yards. Everyone is pleasant, interested and just darn nice. We ~picked~ a GREAT landing neighborhood I tell myself. We locate a decent backyard breach point into the steep ravine by ~our~ tree – the REALLY tall tree, growing from the bottom of a really steep and deep ravine. I’m thinking – oh well, at least we know where it ended up – its in a better place now and at least not lost out to sea.

Imported Photos 00000.JPG

Terry is all like, hey we’re here – lets have a go at it. I protest – as of 8:12am Sparkfun has 66 RockBLOCKs in stock and I am certain they don’t carry Terrys ( Terry does have a brother – but it just would NOT be the same ). I lose this argument – it turns out my considerable voting weight for electronics design and flight control software does not transfer over 1:1 in country for recovery operations votes.  Not even close. Fine.

Imported Photos 00018.JPG

So I proceed to watch our friend Terry ( who I remind you – we have only one of ) climb 75 feet up a tree clearly climbing to his death leaving me to explain to his wife how could I? Oh – and Terry had his truck keys on him to boot. I like taking Ubers as much as the next guy – but NOT to my friend’s house to explain to his wife what he died for – $500 of misc one-off electronics stuffed inside a styrofoam capsule full of telemetry data, pictures of too many clouds and some GoPro video. Oh – and would you mind please giving me a ride back to my car when you are all done crying? I just do not need this today.

Imported Photos 00008.JPG

Terry clearly should have brought extra shoes ( the kind with spikes ). I really should have brought extra pants ( the kind with extra absorption ). You know those moments in time where time seems to stop? You just know something really bad is about to happen and there is absolutely nothing you can do to prevent it? Lets just say it was a really long 2 hours watching Terry climb that tree.  I think I will just stay at ground control next time and get started on the Python and Verilog improvements for next launch thank you very much.

Imported Photos 00032.JPG

To make a long story even longer – Terry survived. He is still with us still today. His two small children still have a father and not a styrofoam space capsule in his place. Colleen can wake up to Terry’s smiling face instead of a solar panel and 1 Hz flashing Navigation strobe LEDs. Terry climbed that ridiculously tall tree ( with safety rope and a carabiner ) and managed to knock the balloon down with a really long ( 15′ ) stick he saw lying in the middle of the really tall tree. It was like watching a Pinata get whacked by a friend on the edge of a cliff over shark infested waters facing certain death and oh by the way – there is no candy to shower down at the end. The styrofoam HAB capsules came down – Jon picked them up, we thanked the nice neighbor Phil ( who took all these great pictures and graciously emailed them to me ) and we were on our way home. Oh, and we did wait for Terry to climb down the tree – he had the only truck keys after all.

Imported Photos 00042.JPG

— Main Story Ends —

— In Closing —

In closing I just want to say how pleased I am with how many accomplishments Team Balloongineers achieved with HAB1 04.27.2016.  For the 1st time we successfully vented excess helium via the Dr.P valve and slowed an ascent ( not enough and not autonomously – but we achieved a major 1st ). For the 1st time we successfully carried liquid ballast aloft and autonomously pumped this ballast to try and slow a descent ( didn’t have a balloon at the time – but still a major 1st – the Flight Computer pumped ballast ). For the 1st time we had semi-reliable downlink telemetry data reporting GPS position, altitude and system status live in flight. Up until descent – we knew exactly in space-time where HAB1 was. For the first time we had a GPS unit capable of working at 100,000 feet and provide accurate position and altitude. For the 1st time we generated electricity in flight with great success above the cloud layer ( 1W at 60,000 feet). For the 1st time we had a HAB capable of night flight per FAA regulations. For the 1st time we demonstrated we could take an actual computer with a file system aloft under battery power and use an advanced language ( Python ) for doing advanced computing in flight real-time like GPS City Lookups, Voice Synthesis, NMEA coordinate conversions, JPEG EXIF information embedding, JPEG text overlaying of telemetry data. For the 1st time we had an advanced power management and fault tolerant “Lizard Brain” FPGA overseer of all operations keeping all systems running. And what I think is the biggest 1st – for the first time, we had remote uplink control of a high altitude balloon 100,000 feet above us in real time. Team Balloongineers typed “open 45” in Issaquah,WA on the ground and at 100,000 up in the air above Mill Creek, HAB1 dutifully obeyed our command and vented excess helium for 45 seconds. Had we instead type “open 90” maybe we would have flown for another day – who knows.

Please enjoys Jon’s fully iMac edited and dramatic GoPro video on YouTube here

Please enjoy the short 6 minute time lapse video made from 5MP RaspPi camera below. Telemetry data was annotated live in-flight by the Pi and is overlayed at the top of each image with time, location, altitude, GPS coordinates, Temperature, Voltage and Current for HAB1. Music performed by the great late David Bowie.

Please enjoy the short 10 minute DashCam video below of the 3 hour flight to 103K feet and the dramatic parachute descent to ground.

 

Post Flight Analysis:

flight_stats.png

All of the retrieved electronics are in perfect working order. The log_gps.txt reported a peak altitude of 31,361 meters ( 102,864 feet = 19.5 miles ) at 00:44:10 Zulu on 04.28.2016. Tons of detailed telemetry data collected of our balloon adventure to near space in the log_primary.txt.

As I suspected, Battery and FPGA were completely golden throughout the flight. Paranoia and insulation was spot on. Hardware Team done good! The flight computer failed to autonomously control ascent however. Software Development and Software Test Teams – maybe some heads should roll over this – oh wait – same head. Never mind!

Post flight calculated ground speed unfortunately indicates we stopped moving once we got above 10km (30,000 feet). At 10km (30K feet) we were doing 60kmh ( 40 mph ) and then above that slowed down to under 10kmh ( 5mph ). Spent 2/3 of the flight at walking speeds – we were going to be stuck in Mill Creek,WA for a while ( normally I would say unfortunately – but my sister lives there – so I just won’t say anything).

Notice the 13 minute gap starting at 00:20:13 Zulu.  CPU has been rebooted due to watchdog timeout with action_satcom(). 7 minutes after the reboot it receives and processed the “open 45” vent command while at 102K feet. The balloon has vented for 45 seconds, but will rise another 1,000 feet in the next 3 minutes to finally burst at 103K feet . The Flight Computer had just rebooted and was unable to calculate ascent after the reboot. Looking at the altitude logs and calculating after the fact, the 45 second helium vent slowed the ascent from 172m/min to 150m/min. Should we have vented longer than 45 seconds? Probably not, autonomous vents are capped at 45 seconds for a reason. Venting longer risks exceeding the 10 minute watchdog timeout and could result in a HAB1 flight computer reboot. Reboots and risky and take time. 45 seconds was the right number. Command was just executed too late.

flight_stats_final.png

We did not travel east as we had hoped to. That was disappointing. There was no wind and we traveled up and hung around Mill Creek for some time before bursting at 103K feet. To be clear, the balloon was going to pop eventually. A weather balloon that starts out at 2m in diameter at launch can expand to a diameter of up to 10m at near outer space. As the balloon climbs to the edge of space it eventually expands to the point where it bursts. We were hoping to delay this burst as much as possible and hover between 70K and 90K under autonomous Flight Computer control – maybe next time. The view at 100K – spectacular.

There was no ONE major failure ( ask any rocket scientist – there rarely are ) . There were a bunch of minor failures that compounded on each other. My one-off HAB1 design includes 2,000 lines of Python for the RaspberryPi Flight Computer and 8,000 lines for Verilog for the Lattice FPGA Lizard Brain. Within these 10,000 lines of source design files I located a single extra “#” present on my part in a battery saving algorithm of the Flight Computer that was removed just days before launch. This algorithm was tested before hand so in the back of my mind I knew valve operations all worked. Unfortunately in removing the battery saving feature, the flight computer also lost the ability to call the open_valve() function.  A single extra “#” pound key ( yes – one key press ) prevented autonomous ascent control from happening. Power On Self Test checks the valve on each powerup, and it still could and did call open_valve() and exercise the valve – so the hardware looked completely functional during pre-flight check. Manual SatCom initiated vents – those worked too. Sigh… extra pounds are just never a good thing in ballooning.

I design ASICs for a living, so this single character within 10,000 lines didn’t surprise me too much. Unfortunate yes – preventable yes – surprising no. Complete failure of the flight computer for autonomous navigation was specifically planned for in advance. We wanted it to work, but always expected it might not. Manual SatCom uplink commands for both venting and ballast pumping could be done – But only if SatCom was working 5×5. At 80K feet SatCom went bad. The worst of the minor failures was definitely the external RockBlock communications thread starving the 10 minute hardware watchdog timer. I never predicted this. Worst case delays for action_satcom() during all the on the ground car “flight” testing was 1 minute max, so 10 minutes for a watchdog timeout seemed the correct setting. At 500 ft/min, a 10 minute delay would result in the flight computer being DOA for 5,000 feet – barely acceptable. A watchdog timeout of 5 minutes would be dangerous given than a single action_satcom() for both uplink and downlink were observed to take 4 minutes on the ground.  More than 10 minutes timeout would literally never fly. The FPGA Lizard Brain did exactly what it was designed to do and ensured the flight computer would be rebooted and brought back on-line in the event of a catastrophic software failure. This happened. The FPGA Lizard Brain Watchdog timer timed out after 10 minutes and the RaspPi was rebooted. Way to go Lizard Brain! This was a huge success, unfortunately the event occurred at the worst possible time at 80K feet during an uncontrolled ascent of 600 ft/min. This would have been a really bad time to lose flight control ( if it had been enabled ). Certainly would like to send a 2nd Pi up just for communications – but that won’t happen ( the extra pounds problem again ).

Do I regret the extra ‘#’ pound?  – of course. Do I question my abilities over it? Hell no. In 6 weeks of spare time I designed and constructed a one-off fault tolerant flight control and communication system that allowed us to manually vent helium from a balloon 100,000 feet above earth – from ANYWHERE ON OR ABOVE THIS PLANET EARTH – how cool is that? Not only did it do this, but it accurately reported position, temperature, voltage and current telemetry data throughout the flight AND correctly recovered flight and communication operations after a catastrophic software failure at 80,000 feet. This desired flight plan failure has actually been a huge success. Fix what is broken, improve and repeat. Hell yeah. Did I mention telemetry data all came in on my Gmail account that I could check from my iPhone while jogging on a treadmill at the gym? Wanna bet even Wernher von Braun would be a little bit envious?

Sharing a single RaspPi serial port and time multiplexing it between the RockBLOCK and the Mesa Bus to the FPGA was a design decision made by me alone upfront to minimize payload weight and power and maximize by limited available time for software development. The ability to develop the Python Flight Control software on a desktop Win7 PC connected to the Lizard Brain FPGA over a single FTDI cable ( via Mesa Bus Protocol ) was a huge win for development schedule. In retrospect – I would design this differently of course for in-flight operations.  The time sharing of a single serial port was functional and tested via hours of convertible drives on the ground -where this all worked flawlessly – but this car driving could not possibly emulate the spotty Iridium connection issues we experienced at 80,000 feet up and no matter how fast I drive my car – I can’t drive 20 miles straight up.

Next iteration HAB2 will add a mini USB Hub ( light as possible ) to the RaspPi A+ and an FTDI cable ( shortened to minimum length for minimum weight ) so that the RockBLOCK gets a dedicated serial port. Any external unpredictable delays from the externally provided RockBLOCK  interface Python module will NOT be allowed to trip up the watchdog timer ( maybe kick the watchdog immediately entering and during action_satcom() instead of just once in the main control loop ? ). A reliable backup recovery system is a must of course – SPOT isn’t it – APRS being the most likely solution going forward as is completely independent of spotting birds in the sky ( APRS is terrestrial 144 MHz 2-meter RF ).

Spending the rest of this weekend examining 3 hours of flight data and 3 days of recovery data to iterate, improve and launch again. Learn from mistakes – improve the design and move forward to the next launch. Man what an adventure. Sign me up for next year ( the electronics and software that is – Terry can keep his tree climbing job ).

What Went Wrong:

  1. RockBLOCK did not work on the descent – AT ALL.  Worst mistake made. Descent was faster than I envisioned and spinning rapidly on the vertical parachute line axis. My car testing with RockBLOCK should have been a clue that descent would be a problem. In my head I had envisioned a nice gently parachute drop – which clearly did not happen.  Watching the GoPro video it is clear that HAB1 not only descended rapidly under parachute deployment but it was rotating as well.  Stuck in the tree, the HAB1 electronics capsule was still spinning. The directional RockBLOCK patch antenna pointed sideways had no chance to establish an Iridium link during the fast and spinning descent.  The sideways mounted RockBLOCK antenna was mandated by the inverted parachute design and lack of a gimble mechanism for all of the electronics. This was a colossal and forseeable failure. Pointing the RockBLOCK patch antenna straight up for both ascent and descent is a must for future HAB missions. A terrestrial backup recovery system such as GPS+GSM Cellular or APRS is needed. I suspect the side mounted RockBLOCK may have been partly to blame for the Watchdog reset as well.
  2. I disabled the ability for autonomous venting. Already tested and working software was modified last minute by commenting out a few lines of Python. These comments were never fully retested. Software Regression Testing is invaluable. One pound over.
  3. The RockBLOCK Python module I imported went away in a CPU no mans land for way longer than I could have ever anticipated – resulting in the watchdog timeout and CPU reboot at a critical moment in flight leaving us entirely in the blind between 80K and 100K feet. If this had not happened, we would have had sufficient time to perform multiple manual vents instead of just the single vent for 45 seconds. This must be fixed. Hopefully not sleeping the RockBLOCK and dedicating a FTDI serial connection to it will do the trick.
  4. SPOT backup GPS locator failed 5 minutes into flight. Backup let us down.
  5. RockBLOCK worked once and only once stuck up in a tree.  It spent 2 days in that tree and sent only a single message out during that entire time.
  6. I tried to conserve Iridium credits and only check for uplink messages every other downlink. Mistake. Latency matters when traveling at 80K feet and ascending at 500 ft/min. Not the time to conserve Iridium credits. Credits are expensive and we planned for a multi-day flight. My algorithm was overly stingy.
  7. Inverted parachute prevented ballast pump from working during descent. The flight computer was desperately trying to pump ballast to prevent descent – but with the inverted parachute configuration, the submersible pump was now at the top of the container and unable to pump anything after a few milliliters. This surely contributed to the RockBLOCK not being able to send any messages on descent.
  8. I was too conservative and stingy in using Iridium credits on the ground for testing.  I didn’t pay for the credits, so I was trying to be a good steward for them and conserve them for the actual flight. HUGE mistake. Should have spent Iridium credits on the ground during car ~flight~ testing like a weekend in Vegas.
  9. Final rescue message at 8:13am reported an FPGA 28bit ON time of 100693 seconds. This is /3,600 = 28 hours. Actual time from Wednesday launch to Friday’s message was 42 hours. This large FPGA time drift still bothers me however. The Twin Paradox could certainly explain a time delta between HAB1’s FPGA RTC and ground control if HAB1’s descent was fast enough ( it was fairly fast ) – but I am fairly certain the chute deployed prior to HAB1 approaching C unless my math is really off and I messed up a few exponents. A more plausible explanation is that the FPGA is using an internal 48 MHz RC oscillator for crude time time keeping for power control, Nav LEDs and serial interface UART baud generation. Actual flight logs all report time using rock solid Zulu time provided by GPS which is accurate to 40 ns ( crazy accurate ).  This needs testing in a controlled ground environment to try and reproduce. Possibly use a real crystal oscillator next go around FPGA pin permitting.

What Went Right:

  1. Decision to switch from Arduino to RaspberryPi was crucial in developing this advanced SatCom capable flight computer in a short amount of time. I am easily 10x to 20x faster and efficient in writing Python versus compiling embedded C++ to a microcontroller. Early software development was done strictly on PC. Final software development was still in a way done on my PC – just via an SSH connection via Ethernet into the Linux based RaspberryPi from my desktop PC ( with the 24″ screen ). RaspberryPi was an excellent choice for this project. Only asked for 100mA – well worth it. Took nice pictures for us too.
  2. Decision to use a FPGA for the Lizard Brain. The Mesa Logic FPGA took care of all the misc IO that a RaspPi’s limited GPIO pins couldn’t do ( FPGA provides 24 GPIOs ). FPGA also handled real time tasks the RaspPi couldn’t handle ( Servo PWM pulse generation, Nav LED 1Hz flashing, real time clock , etc, etc ). This list goes on and on. Chip design is my thing, so designing the critical function in Verilog increased my confidence for the entire project. I am ecstatic that the CPU failed in flight and the FPGA watchdog timer recovered full flight and communications. Well – you know what I mean – I wish it hadn’t failed of course – I’m just ecstatic this fault recovery mechanism worked 100% as planned.  One of my favorite parts of Apollo 13 is:NASA Director: “This could be the worst disaster NASA’s ever experienced.”Gene Kranz: “With all due respect, sir, I believe this is gonna be our finest hour.”  – I feel exactly this way about the HAB-1 RockBLOCK watchdog reset. Things in software failed catastrophically, but HAB1 recovered and could phone home to be recovered.
  3. Decision to dump the barometer and use the U-BLOX NEO-M8N flight capable GPS instead. This unit once configured for flight mode reliably provided altitude and GPS location at 100K feet. Gem of an investment for $35.
  4. Decision to use Mesa Bus Protocol for the link between the CPU and FPGA. Using Mesa Bus Protocol allowed me to prototype software from a Windows PC with only a FTDI cable. Big Win for productivity. Mesa Bus Protocol emulates a 32bit PCI interface – only slower – allowing me to provide a rich 32bit register and SRAM memory space in the FPGA that any CPU could access over a simple serial interface.
  5. Decision to include solar charging. The 3 panels hardly weigh a thing and basically kept the 30,000 mAh batteries fully charged through the flight.  Brought in 1W of solar energy above the clouds. If we manage to float through a day or two on the next flight – this will be crucial in keeping the batteries charged throughout the flight and survive a long wait for recovery. HAB1 batteries were still at 80% after 40 hours in the wild. Be prepared. Winter is Coming.
  6. Decision to log tons of telemetry data to a 32 GB USB flash drive.
  7. Insulation worked fantastic. At -40C outside during the descent, +7C was the coldest any of the electronics got ( unwrapped RockBLOCK ). The batteries wrapped in foil were still at toasty +22C.
  8. Decision to write the last_function() function. This Python function is called whenever a main loop sub-function is called ( such as action_satcom() ) and stores the functions name and time to a file on USB flash drive.  By including this, after a reboot the CPU was able to send back to Earth in-flight that it rebooted and the reason why ( action_satcom() ).  Would have been completely in the dark regarding the reboot otherwise. This little blackbox function was an awesome tool for fault analysis.

Planned Change List:

  1. Place the battery saving algorithm BACK in. I designed (and tested) it for a reason and I still think it was a good idea. In the event of a GPS failure to update altitude ( Very possible at 100K feet ) – the Flight Computer would have been stuck in a loop opening and closing the helium vent valve every minute. There is only one battery and this failure would have brought the entire HAB1 power supply down eventually and prevented it from ever phoning home. The algorithm was a good idea – I should have pushed back harder on agreeing to take it out. It was set for 100 minutes originally – which would have saved the battery in the event of a GPS altitude failure and still allowed complete helium venting. I was the engineer in the thick of it and I failed to push back on upper management when I should have. Lesson learned – be the squeaky wheel.
  2. Change the action_satcom() routine. Allow this to extend the Watchdog timer by kicking the timer inside action_satcom() multiple times. Current design kicks the timer once in the main loop ( which is the proper thing to do ).  With this unknown RockBLOCK failure in the sky – need to try and prevent in from occurring again. Disabling the watchdog entirely is a bad idea – as the CPU could get stuck forever and the FPGA would never reboot it – killing the HAB.
  3. Stop sharing the single Pi serial port. The mux scheme appears to work – but adds risk to the design and also limits the update rate for everything for the entire HAB system.  Given that a separate Flight and Communication computer is not an option ( NASA gets them ) – having a dedicated serial port for each major function is a fair compromise. I won’t say back down on this change. We add  a little more weight for a USB hub and FTDI cable, draw a little more current – but get a much improved HAB system in return. Make SatCom window updates more frequent if possible. Definitely check for uplinks every downlink slot instead of every other.  05.05.2016 Footnote : Have been testing with a USB Hub and FTDI cable for a couple of days now. Current for HAB1 electronics has gone up from 200mA to 300mA with this new setup. HAB2 will increase solar power generation from 1W to 13W using these larger panels that will also form the outer shell for all the electronics. Battery capacity will be reduced (30Ah to 20Ah) to shed weight and HAB2 will be 90% Solar powered during the day and enter a duty cycle hibernation mode at night where FPGA+GPS is powered 100% and CPU+RockBlock is 20%.
  4. A minor thing – RaspPi still images are taken too slowly when interesting things happen ( low altitude, high descent rates ).  Need to improve the photo taking algorithm to take these major events into account. Same for DashCam – although a little more complicated as FPGA controls this autonomously without commands from CPU. It could take suggestions from the CPU. A up/down servo pan mechanism would be great if light enough.
  5. Another minor thing – RaspPi still images are annotated in NMEA  GPS coordinates. Switch to decimal degrees instead ( easier to map ) and also include ascent rates from Nav Computer.  Placing City names on the far right would help keep text aligned for time lapse video post flight also.

Design Details:

HAB1_04_27_2016_Electronics.png

HAB1 electronics are based on a RaspPi A+ running Raspbian Linux and a custom HAB.PY program that talks Mesa Bus Protocol over a single serial link to the Mesa Logic FPGA ( Lattice ICE5LP4K ). By using Mesa Bus Protocol, 100% early on software could be written and tested on a standard desktop PC and connected to the actual hardware using only a FTDI cable. Once the HAB.PY software was ready for “flight” testing ( in my car ), it was copied over to a RaspberryPi. Transition was just a matter of changing some Windows paths to Linux paths.  The Mesa Logic FPGA  interfaces to a RockBLOCK sat modem, a uBlox flight capable GPS unit, helium venting servo, ballast pumping pump and a whole bunch of temperature, voltage and current monitors and low and high side FET switches. The FPGA is the “Lizard Brain” while the RaspPi is the high level Flight and Communications Computer. The entire system is powered from a 5V main supplied by a stack of 3 10,000 mAh 3.7V nominal Lithium Ion batteries that can be recharged in flight via 3 solar panels. Everything combined is less than 3 lbs ( well not including a single extra software “#” pound ). The NMEA decoder actually decodes GPS NMEA streams and stores only valid GPS locked data into a ping-pong SRAM buffer. This allows the CPU to leisurely read the most recent GPS signal rather than having to continuously process the serial stream itself. In battery under voltage conditions, the FPGA can then still receive “last known” position information without the CPU being alive. My only complaint about the uBlox is having to write to it on powerup to configure flight mode.  FPGA power management continuously watched battery voltage levels and throttles back duty cycles to preserve power. For example, then battery drops from 3.7V to 3.6V, the DashCam recorder will only record 1/2 as much as before. Eventually the FPGA shuts auxiliary devices off entirely and begins placing the CPU in a PWM mode of on for 20 minutes, off for 40 minutes. FPGAs main goal is to preserve battery power. Winter is Coming.

image001.png

HAB1 Electronics

mesa_logic_dip.png

Mesa Logic FPGA:

Mesa Logic ( aka Mesa DIP FPGA ) is a general purpose FPGA board designed around the 4,000 Logic Cell Lattice ICE5LP4K-SG48. Mesa Logic is like an FPGA equivalent to an Arduino in that the Lattice JTAG programmer is only needed once after board assembly to load a Mesa Logic bootloader into the PROM.  Once the bootloader is in the PROM, a standard FTDI cable speaking Mesa Bus Protocol  and the .NET BD_SHELL.exe application is all that is needed to load new firmware bitfiles. Mesa Logic provides 24 general purpose LVCMOS 3.3V user IO on a 0.100″ x 0.600″ DIP-32 grid. For HAB1 the FPGA user firmware is the “Lizard Brain” and takes care of power management, PWM Servo control, voltage,current and temperature monitoring and serial GPS decoding of NMEA data. The FPGA decodes and stores the most recent NMEA valid ( GPS locked ) string to SRAM – allowing the CPU to sleep when idle and still have access to most recent valid GPS data. HAB1 system partitioning assigns all the low-level mission critical tasks like power management ( peripheral throttling ) and navigation LEDs to the 100% up and reliable hardware. High level tasks such as communications and navigations are assigned to the RaspPi CPU with only a 2-wire serial communication needed to connected to two talking Mesa Bus Protocol. At OSH-Park the Mesa Logic FPGA boards are located at OSH-Park under BML here. A YouTube video of the Mesa Logic FPGA board under reflow using the BML 1″ SMT Reflow Hotplate is available here:

The Software:

As a BSEE, I haven’t gone by the title “Software Engineer” since 1996 when I switched from typing C full-time for embedded 68K designs to writing Verilog to infer flip-flops and combinatorial logic for ASIC and FPGA hardware designs. For the last 20 years any sequentially executed software I “code” is primarily to support my hardware designs ( ChipVault, BD_SHELL, SUMP, etc ).  I will be the 1st to admit that hab.py is no perfectly engineered piece of software. I am sure there are plenty of internet trolls out there that will rip me to shreds for all the “;” I use in my Python, or my liberal use of self.* globals ( pronounced “Goebbels” as they are in fact truly evil ).  All I can say is, first join an international high altitude balloon team and design part of the system – any part – in 6 weeks – then lets sit down over a beer and I will listen to everything you have to say. I include hap.py as open-source as I hope someone might find bits and pieces of it useful. I am particularly proud of the 20 lines of Python that determines live in flight the closest city in North America that HAB1 is near – all without an internet connection. It was very cool to receive email messages live from HAB1 saying “Bothell,WA” and not have to continually look up GPS coordinates. Others might find all the GPS NMEA translations and JPEG EXIF data embedding useful as well. Just don’t re-use action_satcom() as it still needs a little work.  Talking to Iridium birds 500 miles above is a tricky business. The open-source eSpeak utility is incorporated into HAB.BY to voice over narrate live the DashCam video with altitude and nearest city information. The function  action_cfg_ublox() is golden – as it puts the uBlox GPS unit in flight mode on bootup and configures it to reliably report GPS altitude to 150,000 feet. I certainly wrote more Python try+excepts in HAB.PY then probably all of my other Python scripts combined.  Of course, the failure you don’t plan for ( a 10+ minute timeout to an external RockBLOCK routine ) is the failure that occurs in flight. In closing for this chapter on software – developing Python in a Linux environment was a joy ( stressful but still a joy ). Had my $5 PiZero contained a camera connection – I certainly would have sent it up instead. Definitely never would have made our launch date if the Flight Computer software had been written for my 32bit ARM Mesa-Duino-M0 Arduino-Zero clone as originally planned. Writing Python in a Linux environment is so much more efficient than writing embedded C.

The Custom PCBs

 

balloon_adc.png

The ADC board uses a 74HC4051 to mux 8 analog channels ( 4 temperature, 2 voltage, 2 current ) into a single 12bit ADCS7476 ADC. The FPGA drive S[2:0] with the desired channel to convert and the SPI interface transfers the data. An onboard BU33 3.3V LDO regulates the 5V main to a 3.3V clean supply and reference voltage for the converter.

balloon_thermistor_lmt89.png

HAB1 has 4 LMT89 thermistor breakout boards for measuring outside temperate, battery temperatue, CPU temperature and SatCom RockBLOCK temperature.

balloon_voltage_current_sensor.png

A ZXCT1107 is used for high side current sensing for both the battery and solar panels. This PCB also contains voltage dividers (2:1) for measuring up to 6V from the 3.3V referenced ADC.

balloon_micro_usb_power.png

The Micro USB power board provides for a Micro USB power input. The board was designs to allow diode ORing multiple batteries into various rails but that was not utilized for HAB1.

balloon_n_fet.png

The N_FET board uses 1 or 2 IRML6344 low side FETs for switching on high current devices like the ballast pump and Nav LEDs.

balloon_aux_power.png

The high side FET PCB contains 4 NCP331 high side FETs for switching on power to CPU, Servo and uBlox GPS unit.

IMG_1472.JPG

Mesa Logic 0.100″ (DIP ) next to Mesa Logic 0.050″. These are both multipurpose low cost FPGA boards ( ~$10 ) that I use for BML projects all the time. Programmed using a standard FTDI cable and my BD_SHELL.EXE .NET application written in PowerShell.

IMG_1486.JPG

Very first HAB1 prototype ( yes – Breadboards do still exists and have a use).

IMG_1482.JPG

NavLED testing. The 2W LEDs I got from Osram were blindingly bright. FPGA flashes them automatically at 1 Hz when Solar panels say it is nighttime or one a minute when landing has been declared and HAB1 enters recovery mode.

IMG_1507.JPG

Current and Voltage sense boards. Only 2 of the 3 were used in flight.  All of the HAB1 PCB designs use castellated vias on a 0.100″ x 0.600″ DIP grid. Using castellated vias allows me to design a single large board in the future ( HAB2? ) that could interconnect everything. For HAB1 – I just soldered a whole bunch of cable harnesses – which made the entire design very scalable and flexible.

IMG_1528.JPG

Simple power resistor for current limiting the LEDs. These boards are sandwiched between the batteries and use excess waste heat to keep the batteries warm.

IMG_1545.JPG

Battery module containing 2 10,000 mAh 3.7V Lithium Ions, the original solar charger and USB switcher board and a dedicated 5V switcher  I purchased that doesn’t turn off ( unlike the one that came with the battery ). A 3rd battery rests underneath the RaspPi A+.

IMG_1544.JPG

The 5V DC/DC I purchased couple with my voltage and current sensor board and a high side FET board. The DC/DC has a LED that turns on when the battery is above 3.5V. This is used to turn off the FET board when it drops below 3.5V.

IMG_1539.JPG

The custom electronics on top of the RaspPi A+ plugged into the 2×20 GPIO header on top. During prototyping, the 22 gauge wire was just plugged into the SIP sockets on each board. After pinout was finalized, I soldered all the wires in.

IMG_1565.JPG

 

This is what it looks like when you run out of time to design a final interposer PCB and instead connect up all the misc PCBs with wire. Yes – it all worked – even at 103,000 feet (31km).
IMG_1566.JPG

The top is sealed with the lid.  The wires all come out a single slit on the side that is eventually sealed with kapton tape.

IMG_1484.JPG

Testing the solar panels ability to charge a dead battery.

IMG_1479.JPG

Testing the RockBLOCK from my house the day it arrived.

IMG_1480.JPG

Notice the number of failures before the 1st message gets through.
IMG_1491.JPG

All three modules plus cells with all the cable harnessing to interconnect them. Original mechanical design for HAB1 called for a 1/2″ wooden dowel to run through the center of the sphere – which mandated the three separate tupperware containers into of just a single large container.  We ditched the rod and used duct tape instead ( which worked great ). I already had all the cable harnessing done, so kept the three containers. This supported wrapping the battery module and electronics flight computer module in foil for heat retention, alpha particle shielding (rad hard) and also RF noise shielding. RockBLOCK had only the sealed air environment of the tupperware container to retain heat. This can be observed in telemetry thermal readings throughout the flight.

IMG_1580.JPG

Testing the ballast pump.  That little 3V aquarium pump was quite the find. It emptied this entire container in 30 seconds.

IMG_1572.JPG

Quick road test. GPS works great – RockBLOCK – not so much. I switched to doing road tests in my convertible after this – which was slightly better.

IMG_1587.JPG

Everything wrapped up and ready. Notice all the foil – especially the long 12″ RaspPi Camera ribbon cable. Giant RF emitter that thing is. RockBLOCK is an amazing little device. The Iridium Satellites are in low Earth orbit at a height of approximately 485 mi (781 km). RockBLOCK on the ground – this little unit, talks to 1 of 66 satellites (some of the time) almost 500 miles away. Getting above the clouds helped tremendously, but 100,000 feet is only 20 miles. Those birds are still way – way up there. RF shielding of all the other electronics was very important. The grey blocks are foam cut up into “cheese blocks” and stuffed in all the extra space. Not so much for insulation, but to keep everything from shifing when the balloon bursts and everything is violently inverted for the parachute deployment. Having all the electronics working during descent was crucial for phoning home landing location.

IMG_1499.JPG

1st pass at the electronics all in a single sphere ready for launch.  My teammates liked the Muppit look. Jon named her Cloey. I ditched the outer foil in the end for fear of blocking Iridium signals.  HAB2 is currently in the planning stages. HAB1 is signing off. Thank you HAB1 for all your valiant efforts in getting home.

Thank you for reading my blog. This was quite the 6 week adventure from greenhorn team member to reaching near space and returning to tell about it.

The End.

In The Press

Max ( Clive Maxfield ) author of this hilarious book on electronics and also EE Times columnist wrote this article on the HAB1 journey and BML.

Richard Baguley of Hackaday wrote this article on the HAB1 journey.

Lisa Martin of Makezine.com wrote this article on the HAB1 journey.

 

 

HAB1 04.27.2016 Ascent to 31K Meters (103,000 Feet)

1″ 100W Hot-Plate for SMT Reflow

IMG_1453.JPG

[03.13.2016]

Black Mesa Labs has been using a $20 hot plate for a year now for soldering QFN ICs to PCBs using the BML Inverted Solder Ball reflow technique. This works great for soldering 0.5mm pitch QFN and LGA type packages that would otherwise require messy stencils and paste. Only issue so far has been the size ( 10″x10″x3″ ) and thermal mass of the commercial hot plate as it consumes precious microscope work area and unfortunately stays quite hot for 30+ minutes after a quick 4 minute single IC reflow job.  BML boards are mostly 1″x1″, so a 800W hot plate with a 6″ diameter heating surface is overkill for most jobs.

heating_element.jpg

Wanting something much smaller for a typical BML PCB – stumbled across this 24V DC heating element on Amazon for only $14. It is rated for 24V at 5-7 ohms ( or 4.8Amps ).  A surplus 19.5V DC 5A laptop power brick laying around BML seemed like a perfect match for this element.  BML has safety rules avoiding designs above 48V – so the 100Watt 20V DC supply coupled with the 24V element seemed like a great way to make a lot of heat in a small surface area in a short amount of time.

IMG_1449.JPG

First experiment was to see if this little element would reach +200C in under 4 minutes. Not wanting to cut up the laptop supply’s power connector – located this

71CtDnlyQRL._SL1500_.jpg

power connector adapter for only $5 and cut off the yellow Lenovo end of it and converted it to 0.100″ spaced Dupont connector instead. Hooked up a simple push button switch in line with the circuit – pressed the button and measured surface temperature with a digital thermometer – reached 200C in only 90 seconds!  Too fast for reflowing 40nm FPGAs unfortunately. Wanting to keep the 20V supply – needed to switch to pulse width modulation to slow down the thermal ramp to closely match solder reflow profile of 4 minutes to +200C. A relay? Nah – lets design a custom PCB with power FETs!

pcb.png

After consulting with BML’s “all things analog expert on retainer” DP on how best to switch 100W at 20V – designed a small $2 2-layer PCB as a low-side switch using two IRLML6344TRPBFCT-ND 30V Power MOSFETs in parallel.  Design was drawn ( no schematic ) straight in CopperConnection Gerber drawing tool in less than 15 minutes and fabbed by OSH-Park in about 2 weeks for $2 for a ~1″ x 0.5″ 2-layer 0.8mm PCB.

IMG_1454.JPG

Rds On for this N-FET is 0.029 ohms at 5A ( or  725 mW ) which is near the 1.3W limit for the itty bitty SOT23 device. Not having any heatsinks other than the connection to the PCBs giant “Drain Plane” – opted to share the 5A load across two FETs in parallel. The N-Channel MOSFET’s Source is technically ground – but all the heat is dissipated through the Drain, so large pours on both PCB top and bottom were made to better dissipate the Rds On heat. Design is feed thru for + and – from the supply to the + and – for the load even though the + supply doesn’t touch the circuit at all.  Placed 3 different sized vias 0.125″, 0.079″ and 0.040″ so the board may be easily reused for other projects of various gauge wire needing high voltage and current switching.  A 100K resistor to GND ensures the FETs don’t turn on by themselves without a driver present (powerup Hi-Z) and a 1K series resistor to the Gate of both FETs limits the in-rush current. These Infineon  N-Channel FETs are only $0.29, come in a small SOT-23 package, can switch 5Amps from a 3V LVCMOS signal and are now BML’s go to device for power switching.

20131019_345545.png

Above is a Owon Scope capture of the FET turn on time ( Gate versus Drain ). CH2 ( Yellow ) is the Gate of the N-FET which is driven by LVCMOS 3.3V totem-pole from an AVR through a 1K series resistor. It takes 500ns for the Gate to go from 0V to 1V and then the FET turns on.  CH1 (Red ) is the Drain transitioning from pulled up ( via 1K thru the LED ) to clamping to GND. This transition takes about 2uS total.

10914-01a.jpg

 

An Arduino Pro (above) was chosen as the PWM controller.  Most BML CPU designs are either RaspPi or Arduino-Zero based ( ARM 32bit ), but BML had this old Sparkfun board in a box and it was the right dimensions as a mechanical base for the project. The 1.5″ diameter heating element is suspended about 1″ in the air, so the wide 2″x2″ Pro board keeps everything from tipping over.
IMG_1451.JPG

Above picture shows everything cobbled together. A small piece of aluminum foil is wrapped around the element to keep flux from making a mess of it while under reflow. The heating element comes with 22 AWG solid wire crimped to it and extending about 2″ away. The wires are soldered to a 1×2 SIP connector which then mates to another 1×2 SIP connector on the FET board ( being able to remove the heating element from the circuit while under development was VERY important ).  The FET board is placed in the center of the Arduino Pro board atop some poster putty and held in place with the Enable wire (White ), the 1,500 resistor voltage drop network and a ground wire. The Pro can be powered from an FTDI USB cable of course, but to power it from the single Laptop supply, the voltage required dropping down from 20V to around 12V for the on board LDO.  Three 500 ohm leaded resistors each provide about a 2V drop while the AVR CPU is sipping 4mA, for a total drop of around 6V. A large 1K resistor provides 20mA to the red LED whenever the FET gates are turned on and the heating element is getting power.  Total cost was under $20 for the project – not counting the Arduino Pro in the junk box.MLX90614(E,K)SF-xxA.JPG

The Arduino was chosen over an FPGA as the PWM rate is really slow ( seconds ) and the Arduino makes it easy to expand the design and eventually interface to a Infrared Thermometer over I2C via existing tutorials and libraries.  Eventually the design could be expanded to measure the temperature at the surface of the FR4 under reflow for either reporting and / or control.  For now – the dead reckoning PWM method is plenty sufficient.  As the heating element contains very little thermal mass – it cools down in just seconds.

Above is the target solder reflow profile and below is the achieved profile as measured with a thermocouple on a scrap 1″x1″ FR4 PCB.

The GPL’d open source C++ source is available here hot_plate and is fairly straight forward.  When the button is pressed, the AVR starts a 4 minute PWM state machine that gradually increases the PWM duty cycle from 30% to 40% to 60% and then off.  Measuring a scrap piece of FR4 with a thermocouple mounted to it, an ideal reflow profile of 20-150C for 90 seconds, 150-200C for 90 seconds and finally 200-250C for 50 seconds was achieved. Pressing the button mid thermal cycle cancels the cycle. The AVR and giant Pro board is definitely overkill – but the PCB size made it a great platform to mount everything else to.

A word on safety – although this project is under 48V – the amount of heat it can generate could definitely start a fire if powered up in the wrong place ( under a stack of old newspapers for example ) .  To prevent any potential small fires from starting 3 levels of safety were built it. 1) the Arduino runs the heat profile only when the pushbutton is pressed and then automatically turns itself off after the profile time is complete. 2) The 20V 5A power to the electronics goes through a toggle switch. 3) The original laptop barrel connector is quickly and visibly disconnected from the setup when reflow is complete allowing the hotplate to be visibly disconnected from the power supply when not used.

Below is a YouTube video of a Lattice ICE5LP4K-SGN48 FPGA reflowing on the “Mesa Logic DIP” PCB using the BML Inverted Solder Ball reflow technique. This is a general purpose FPGA board on a 0.100″ grid providing 24 user IOs in a 0.600″ x 1.6″ package. This board has a destiny to reach 100,000 feet (~30km) this coming May as a HAB controller for a weather balloon. It will communicate with both an “on balloon” Raspberry Pi and with a ground station via a RockBLOCK Iridium satellite modem using Mesa Bus Protocol. Stay tuned to Black Mesa Labs for an upcoming blog on all the details of the custom HAB electronics.

 

 

1″ 100W Hot-Plate for SMT Reflow

Mesa Bus Protocol

Black_Mesa_logo.svg

Enclosed text file ( mesa_bus  ) is the open-source  Mesa Bus Protocol for transferring bytes between CPUs and FPGAs. It is intended to transfer data between 50 Kbps up to 10 Gbps over UART to SERDES links with just a few wires and very little hardware overhead. It is a very small gate foot print byte transport protocol for encapsulating and transferring payloads of higher protocols (0-255 bytes per payload). Think of it like a cross between Ethernet and USB 3.1. The advantages Mesa Bus Protocol has over USB, Ethernet and PCI is that it fits easily within a $3 FPGA and may be bus mastered either by a PC with a FTDI cable or any old ARM/AVR Arduino CPU (or RPi) with just two standard UART serial port pins. As it is ASCII based, Mesa Bus Protocol is also very portable to wireless devices such as Bluetooth SPP and the RockBLOCK satellite modem for the Iridium network .

envelope2

Have you ever placed a regular envelope inside an interoffice envelope? Mesa Bus is that interoffice envelope. Where Ethernet transfers UDP, TCP, HTTP,etc packets without higher level knowledge – Mesa Bus can transfer SPI, I2C, Digital Video, 32bit LocalBus,etc  from chip to chip without protocol knowledge.  Mesa Bus works with up to 254 devices with only 2 wires between each device. Black Mesa Labs currently uses Mesa-Bus with the Mesa-Logic MCM FPGA and BD_SHELL.exe windows executable, Python, or an Arduino-Zero ( ARM ) as bus master. Payloads have been digital video ( 800×600 at reduced frame rates over a 40 MHz synchronous Mesa Bus link ) and 32bit local bus write and read cycles (PCI-ish) to FPGA registers and also for FPGA flash firmware updates. Mesa Bus Protocol fully supports prototyping embedded software 1st on a PC in a rapid scripting language like Python as it is clear ASCII that may be transported over a standard FTDI cable or Bluetooth SPP connection.

 

 

 

 

Mesa Bus Protocol

Quick and Easy Fence Post Replacement

IMG_1377

Taking a short break from electronics to document an easy solution for the frustrating home repair of rotten fence post replacement.  Here in Seattle, it is quite common for pressure treated 4×4 fence posts to rot from the ground level down after about 20 years in the humidity. Standard replacement method is to dig out and smash up all of the 24″ concrete base surrounding the rotten 4×4 and replace with new concrete. Two feet ( ~1/2 a meter ) down is a lot of digging ( especially in Seattle clay+rock soil ) and with 3 rotten posts ( and 27 more most likely to fail in the next 5 years ) – I set out on an engineering mission to find an easier – greener – way. This might not seem appropriate for a Black Mesa Labs post – but just like how BML does things very different in electronics from society norm ( example : soldering 45nm QFN FPGAs onto 2-layer PCBs using a $20 hot plate ) – the fence post removal was a full Thomas Edison obsession of doing something different through persistent experimentation. Going into this project I was told by countless people 1) That just isn’t how it is done, 2) It won’t work and 3) Just hire a guy to take care of it. Like all things at Black Mesa Labs,  I was determined to do things different.

Through much trial and error – this is the Black Mesa Lab “Cork Popper” method of fence post removal and replacement without all the digging. This method supports removing the rotten section of 4×4 while retaining the majority ( 75%) of the existing concrete base in place. Keeping the existing concrete also means the new post will be in the same location and plumb as the previous post. Less waste, less effort, faster repair.

Step-1 ) Tear down the fence sections supported by the rotted 4×4 post. A Sawzall with a metal cutting blade that rips through nails makes quick work of this.

 

Step-2) Dig a hole 12 inches ( 1/3 meter ) down around just two side of the concrete base. This is only 1/2 the depth of the entire concrete base. Once exposed, this concrete will now be like an egg-shell ( assuming it has been in the ground for 20 years ). The packed earth is the only thing holding this structure together at this point in time. Using a wrecking bar and a small sledge ( or even just a hammer ) – pry off the exposed concrete. It should easily break off with very little effort.

 

 

Step-3) Post Extraction.  With the concrete removed, the post is now much easier to extract as only the very bottom 12″ is applying pressure against all 4 sides.  If the post is not entirely rotten, the easiest method is to drill a 3/4″ hole through the post above ground level and shove a segment of 1/2″ black iron pipe.  For less than USD $40 total I purchased two matching 6 ton hydraulic bottle jacks which can then lift the iron pipe ( and 4×4 post ) slowly out of the ground.

IMG_1369

Step-4) Is only necessary if Step-3 failed – the post is so rotten that it breaks into 2 pieces, a 6′ above ground section and a 2′ below ground section.  Using a small hand saw ( such as this great little folding jab saw ) – or an awesome mini “Sawzall” such as this 12V Bosch reciprocating saw cut the rotten section away and leave a nice level 4×4 surface of fairly decent wood. A friend of mine says this 12V Bosch is actually made by Playskool , but I don’t care – it gets into really tight places. Trick is to have lots of 12V Lithium Ion batteries ( I have 5 ).

 

Step-5) Attach the extractor. I built this extractor for about $40 using these 12″ lag bolts and some eye bolts from Home Depot and this 5,000 lb 36″ trailer safety chain. I have 30 posts total in my fence – so money well spent. Extracting the old 4×4 in tact means I can reuse 75% of the existing concrete base. This is about 150lbs of concrete I don’t need to buy, haul and mix and old concrete I don’t need to dig out of the ground and toss into a land fill.  Win-win.

IMG_1376

Step-6) Screw the extractor into the remaining section of post using a ratchet and then lift it out using the bottle jacks.

 

Step-7) Insert the new post. I used Hem-Fir Severe Weather Standard Pressure Treated Lumber (Common: 4 x 4 x 8; Actual: 3.5625-in x 3.5625-in x 96-in) from Lowes for $10. The 4×4 post should go in easily assuming dry wood and should already be plumb ( but check with a level ). Pounding with a 4lb sledge from the top if necessary. I really like this 4lb sledge as it is small enough to wield on top of a 6-foot ladder like a normal hammer but still packs a punch. For the old/new hybrid base, surround the exposed 4×4 with some chunks of 4x4s or 2x4s as temporary spacers and then surround those with some cardboard. Fill in outside with packed dirt. Remove the wood blocks and pore the concrete between the post and the cardboard. I used a single 80lb bag of concrete for 2 posts. I think next time I will buy 50lb bags instead as the 80lb bag is just awkward to handle. While the concrete sets – remove old nails from the fence panels and then start rebuilding.  I rebuilt using these Deckmate star drive screws and my Bosch Impact Driver. Very little effort and time compared to hammering and will make the next tear down job much easier. Hopefully this post will last another 20 years.

12 hours of work start to finish and the Job is done!

I hope you found this short tutorial on a quick and easier method of fence post extraction and replacement helpful.

Quick and Easy Fence Post Replacement

The Future is Now!

IMG_1218 - Copy (2)

09.16.2015 – Black Mesa Labs was delivered a very unexpected large 2’x2′ (60×60 cm) package today. BML generally uses a U.S. quarter as a scale reference – but for this – a Pint glass is in order:

IMG_1209

Examining the shipping label – Realized the new Lattice ICE-Ultra FPGAs had arrived 3 weeks early. This new 40nm FPGA from Lattice is a game changer for both BML and the Maker Movement.

IMG_1222

Backstory – Back in May 2015, BML started experimenting with Lattice ICE40 family  as an alternative to large Xilinx TQFP devices for BML designs.  The ICE40LP384-SG32 appeared to be a game changer for BML as it was both low cost ( sub $2 ) and in a 2-layer PCB friendly package ( SG32 ). The big two FPGA vendors ( let’s call them Brand-X and Brand-A ) have gone entirely BGA for their FPGAs – which is completely useless for BML and the OSH-Park style 2-layer Maker Movement PCB designs and assembly. Lattice ( although offering mostly BGA ) has FPGAs in QFN packages ( aka MLFs, SGs ) which can be hand soldered. Sadly – the LP384 proved limited in capabilities – a simple Mesa-Bus bridge design that would easily fit in 200 Xilinx LCELLs completely filled the LP384 as 1/2 the LUTs were used as route-throughs ( consuming limited gates for simple routing ). The LP384 turned out to be about as useful as a large CoolRunner CPLD. But wait….

In June 2015 – Lattice announced that their newer ICE-Ultra ICE5LP4K ( 4,000 Logic Cells ) that was originally BGA only would soon be available in a SG48 package (QFN).  4,000 FPGA Logic Cells for $5 in a 7x7mm package that is 2-layer PCB friendly –  Sign me up!  Fast forward to July – the ICE5LP4K-SG48s were nowhere to be found in normal distribution.  BML contacted both Digikey and Lattice and learned the devices would need to be special ordered in quantity in uncut reels – with a minimum reel count of 50, and a minimum order of 10 reels. The ICE5LP4K FPGAs themselves are only $5, but 10x50x$5= $2,500 – not exactly within BML $100/month beer money budget.  Through persistence and careful negotiations, BML was able to secure a “Special Order” for a single reel of Qty-50 FPGAs – or $250 – manageable. Placed the order in July with an ETA of October 7, 2015 – but they arrived 3 weeks early – Christmas in September!

Inside the giant box was a little LP vinyl sized box and a whole lot of bubble wrap:

IMG_1211

Inside the little box, the giant reel:

IMG_1213

Unfurling the reel – the actual chips – this is all fifty chips:

IMG_1215

Single chip- 4,000 Logic Cells in 7x7mm package for $5 – how cool is that?

IMG_1218 - Copy (2)

Backside you can see how it is 2-layer friendly unlike BGA devices:

IMG_1219 - Copy (2)

This FPGA is really a game changer as it is a true FPGA, much more than just a simple CPLD.  It contains 80 Kbits of SRAM, an on chip ~48MHz oscillator, 39 LVCMOS IOs and 3520 Logic Elements ( LUTS + FlipFlop ). A close Xilinx equivalent ( in a much larger 14x14mm TQFP  package ) would be a device like the Spartan3 XC3S200. This new Lattice device is 1/4 the size and 1/3 the price and 2-layer PCB compatible. 4K LEs aren’t enough for a custom CPU or DSP work – but ample logic for bus bridging, UART designs and complicated FSMs ( Finite State Machines ).  My general rule is 100 LEs a designer can easily consume in a short day of RTL design, 1,000 LEs in a week, 10,000 LEs in a month. 4K LEs is a perfect canvas size for BML designs.

Next up is assembly. BML has 3 PCBs that were waiting for these devices to arrive, Mesa-Video, Mesa-Logic and Mesa-GPIO:

IMG_1221

Initial use for this chip is the Mesa-Bus bridge function – a daisy-chained serial bus that self enumerates based on location in a serial chain and converts clear text ASCII UART serial into either SPI or I2C. Purpose of Mesa-Bus is to provide USB like expansion to simple microcontrollers like Arduinos using only 2 pins ( TXD and RXD UART Serial ). Stay Tuned – the family of Mesa-Modules are about to come alive!

The Future is Now!

Mesa-Video : 800×600 Digital video for Arduinos over 2-wire serial Mesa-Bus

Mesa-Video : 800×600 Digital video for Arduinos over 2-wire serial Mesa-Bus

[ 08.30.2015 ]

IMG_1174

This post describes Mesa-Video, a low cost, low power, small size and fully Open Source Hardware and Software solution for providing 800×600 digital video for Arduino ( and other ) microcontrollers.  Mesa-Video makes it quick and easy to display text and 24bit color graphics from any MCU using a single UART serial port pin. Applications for Mesa-Video are embedded projects requiring video output and embedded developers wanting real time visibility into their system operation. Mesa-Video is the 1st of multiple Mesa-Modules planned. For a quick summary of Mesa-Video and Mesa-Bus, please read the very nice article at Hackaday by Richard Baguley from 09_01_2015.

IMG_1100

[ GPU SPI Bus Interface ] On the original EVEy Video board  ( shown above ) the native SPI interface for the FT813 GPU‘s is brought out to a “Nano Bus” header which was then bit banged with Python software on a PC going through a BML Nano3 FPGA (shown below) containing simple GPIO logic. This was perfect for initial board test and GPU bring up, but slow and needing a faster interface to PCs and micro controllers for a final product.

IMG_1129

[ The Arduino SPI Problem ] : A major problem with SPI ( and I2C ) on Arduino is that with shield stacking it is difficult to get more than a single shield device on the bus. SPI devices all require a unique chip select and I2C devices all require a unique address. This is all a bit like the pre-PnP era of plugging ISA cards into a DOS PC and having to select IO Port Number and Interrupt line jumpers. Stacking Arduino shields isn’t exactly as Plug-and-Play simple as plugging in  USB peripherals or PCI boards into a PC. A better solution that scales like USB for Arduinos is needed. Black Mesa Labs has created a simple and low cost solution for this called the “Mesa-Bus”.

[ Introducing the Mesa-Bus from Black Mesa Labs ]

Mesa-Bus

The “Mesa-Bus” is a simple and completely open text serial protocol designed for transporting self enumerating SPI and I2C bus traffic over a standard UART serial connection. Electrically the interface is simply LVCMOS 3.3V RX and TX serial on a standard FTDI 1×6 6 pin 0.100″ header. Why UART Serial? Every device from PC to Arduino to 8051 has at least one UART serial port. Arduino’s are especially flexible as with Software Serial any unused IO pins may be turned into an additional serial port. The Mesa-Bus is made possible by a custom BML FPGA bridge design that has an autobauding UART, auto-enumeration logic and a SPI/I2C bridge residing in a low cost ICE40 Lattice FPGA. Why use the FTDI 1×6 0.100″ connector? The 3V FTDI 6pin connector is ideal as it is well known, bread board friendly, provides raw 5V for powering modules and has 4 signal pins, 2 Ins and 2 Outs. The Mesa-Bus utilizes all 6 pins for allowing Mesa-Modules to be easily powered and configured either by a Arduino like MCU or a full fledged PC with a $20 FTDI serial cable. Aren’t UART serial connections too slow? The Arduino-Zero example for Mesa-Video development communicates at 3Mbaud over a single wire, considerably faster than I2C and in the ballpark of traditional 4 pin SPI using just 2 pins on the Arduino.

Mesa-Bridge-FPGA

This small $3 Lattice ICE40LP384 FPGA design below (22x32mm) bridges the “Mesa-Bus” to SPI and allows the EVEy-Video board’s FT813 GPU to interface either to a PC or an Arduino board via the FTDI 1×6 header. The UART sub-component of the FPGA design autobauds to the 1st “\n” character received from the Bus Master and uses that baud rate for upstream ( RDo ) and downstream ( WRo ) communications – allowing Mesa-Bus to scale from high speed 32bit ARM serial ports to simple 9,600 baud bluetooth wireless virtual serial ports. The BML custom Enumerate and Decode Logic inside the FPGA performs bus enumeration so that each device in a Mesa-Bus serial chain is uniquely addressable to the host ( similar to USB enumeration at a high level concept ).  The Bridge Logic bridges the clear text ASCII characters to SPI ( or I2C ) bus cycles specific to the slave device’s requirements on the module. The final Mesa-Video design merges this bridge FPGA design along with the GPU and HDMI converter onto a single board.

ICE40LP384

[ PC to Mesa-Bus ]

Why support interfacing Mesa-Bus to a PC that already has USB? Having worked on embedded software+hardware development for 20 years, I have found it tremendously beneficial to be able to prototype embedded software 1st on a PC with a scripting language like Python and then port the software to C for the final embedded MPU. MesaBus is designed to support this development flow.  All Mesa-Video software development was done in rapid prototyping using the Python scripting language on a PC. Once all the initial bringup and custom functions were working on the PC, the software was quickly ported from Python to Arduino C and working stand alone from an Arduino Zero board. This flow is a tremendous productivity enhancer as it avoids the compile, download and repeat loop that often needlessly consumes engineering time developing embedded systems. Prototyping in Python also makes software easily portable to RaspberryPi platforms.MesaVideo(1)

What is the point of “Mesa-Bus” – why not just use SPI natively? “Mesa-Bus” aims to solve both the hardware and software complexity of integrating multiple SPI and I2C peripherals by using any readily available 2-pin serial port and transferring everything in ASCII clear text for easier documentation, development and integration. Binary protocols are more bit efficient, but dealing with clear text protocol transfers is much more human efficient in terms of documentation, development and debug ( think XML versus proprietary binary file formats ).  With Mesa-Bus there are no clocks edges, chip selects or I2C addresses to worry about. All Arduinos have at least one serial port and a virtually unlimited number of Soft Serial ports. MesaBus solves the problem of limited number of available Arduino pins ( think 8-pin ATtiny  ) by allowing for device daisy chaining with built in device enumeration. Mesa-Bus isn’t multi-drop ( like RS485 ), but point-2-point like Token-Ring, electrically superior for performance ( less capacitance, shorter wires ). Multiple MesaBus modules connect to a single Arduino ( or other MCU ) serial port and are uniquely addressed automatically based on their location in the chain. Modules are breadboard friendly with their 0.100″ 1×6 male headers and a 4 slot ( expandable to 8,12,etc ) back plane is on the drawing board for compactly connecting multiple Mesa-Modules in a single serial chain in a 1″ x 2″ form factor.

Back to Mesa-Video – here are the two separate boards joined together.

IMG_1197

The final PCB iteration combines both designs onto a single 1″x3″ (3cm x 8cm) board with no center connectors for the SPI interface.

Mesa-Video

[ Video Capabilities ] : Mesa-Video software is a subset of the entire FT813 GPU video capabilities in order to make it quick and easy for novice to expert Arduino developers to display text and limited graphics from their Arduino onto any HDMI ( or DVI ) display. The FT813 GPU is extremely capable, but also complicated and non trivial to get started programming with the entire API from FTDI. Mesa-Video simplifies things by providing a simple UART serial interface on the input and a fixed 800×600 HDMI interface on the output. MesaVideo is intended for displaying real time information ( like live variable status, progress bars, stock quotes, centrifuge status, etc ). Why not just use a RaspberryPi? RPis are great – but they aren’t microcontrollers – they are full fledged Linux workstations with powerup and powerdown requirements ( and memory card corruption issues ) that Arduinos just don’t have to worry about. Mesa-Video fills the niche of when a small Arduino is the right tool for the embedded job ( flying quad-copter?), but a graphic output ( either just for development debug or final product ) is needed.

IMG_1154

Below is an example electrical setup of Arduino ZeroPro + Mesa-Video illustrating the advantage of Mesa-Bus over a conventional SPI interface. There are only 3 wires, +5V, GND and TX ( Pin-1 in Arduino parlance, WRi on Mesa-Bus ) connecting the Arduino to Mesa-Video. Mesa-Video is a write-only interface so the RX serial pin doesn’t need to be hooked up.:

IMG_1186

First “Hello World” Example Sketch for Mesa-Video. There are no include files, this is everything that is needed to bring up the GPU and display basic text at a specified (x,y) location. The hex strings within brackets are actually SPI bus cycles. Encoding the SPI bus cycles in ASCII text makes Mesa-Bus designs portable, easy to document and highly compressible ( as shown ). The low cost FPGA on each Mesa-Module does all the required conversion from ASCII text to SPI ( or I2C ):

arduino_ide

Below is an out of focus picture of the Hello World sketch ( actual text on the LCD looks great – the digital HDMI SVGA signal is perfect ). Basic ( non-painted ) text is available in 1x, 2x or 4x sized VGA fonts:

IMG_1191

[ Whats Next ] :   Next step is to assemble the final board that merges the Mesa-Bus SPI bridge FPGA and the GPU+HDMI onto a single board.  Layout of the merged design is shown below in CopperConnection – a fantastic simple to use and low cost layout tool that exports Gerbers ready for OSH-Park.  CopperConnection is as easy to use as ExpressPCB – but it generates Gerbers that can be fabbed out anywhere. Best $50 I’ve spent in a long time. I’ve since upgraded to the $100 “Ultimate” edition that allows me to generate and hand edit ( via custom Python scripts ) the actual board files. Fantastic Tool – check it out over at RobotRoom  free version doesn’t export Gerbers – but is a nice demo.  Below is the merged Mesa-Video design, it is 1″ x 3″ ( 77 x 26mm ).

mesa_video_cc

The FPGA has been upgraded from the 400 Logic Cell ICE40LP384 to the new 4,000 Logic Cell ICE5LP4K. The new and larger FPGA is a game changer for BML as it is only a few dollars more and provides 10x the resources, contains 80Kb of SRAM and also an internal 48 MHz oscillator in a 1cm^2 package that is 2-layer friendly.  To simplify design and manufacturing, the single ICE5LP4K device will be used on all Mesa-Modules going forward. The larger FPGA with SRAM supports an alternative bitstream idea that could potentially turn Mesa-Video into a 4-channel LVCMOS digital logic analyzer ( no computer involved ). The power of open-source is that it turns any project like Mesa-Video into a canvas for others to improve upon. Final firmware design will be EEPROM upgradable over the Mesa-Bus interface – supporting in-field FPGA upgrades from any PC with a FTDI cable.

IMG_1193

[ Mesa-Duino-M0 ] : Mesa-Duino ( for short, pictured above – 1st proto ) is an Arduino-M0 compatible derivative that plugs directly into slot-0 of the Mesa-Backplane and becomes the bus master instead of an external board. Total BOM for the assembly is less than $10 – which is pretty impressive for a 48 MHz 32bit ARM CPU with 256KB Flash and 32KB SRAM in a 1″ x 1″ form factor.  The actual Arduino-Zero (.cc) and Arduino-Zero-Pro (.org) contain an Atmel-ICE on-board. The Mesa-Duino design removes the ICE and makes it just the ARM CPU, a FTDI connector for the Mesa-Bus mastering and a micro-USB for firmware upgrades – reducing the BOM item count by like 90% while maintaining compatibility with the Arduino IDE ( ultimate goal – and why ATSAMD21G18 was chosen ).  BML encourages users to use whatever microcontroller they already have for Mesa-Bus mastering – but if you wan’t something really powerful, small and inexpensive – the Mesa-Duino with a 3Mbps hardware serial UART on board is a nice solution. Next board spin will remove the large 2×8 header. If BML can locate someone to manufacture the Mesa-Duino, the intention would be to provide kick back royalties to the Arduino organization for future Arduino development expenses.

[ PCBs also in the queue ] : Along with Mesa-Video, 4 other PCBs have been sent out to fab to begin the Mesa-Module family. All Mesa-Modules heavily reuse the same FPGA layout and are all 1″ wide with identical 1×6 0.100″ right angle header connector for plugging into a FTDI Cable, a Mesa-Backplane or a standard bread board. Cost is very low, with BOM for all below $10.

Mesa-Logic : A general purpose FPGA development board with 4,000 Logic Cells, SRAM on chip and 16 LVCMOS 3.3V IOs ( limited drive strength). The FPGA will be firmware upgradable in the field allowing users to design whatever.

mesa_logic_cc

Mesa-GPIO : A general purpose IO module with 16 3V 25mA drivers, 5V tolerant via a PCA9539 IO Port Expander IC. 1.5″x1″ (38x26mm). Mesa-GPIO is a typical example of a low cost Mesa-Module, left 1/2 of the PCB is the Mesa-Bus 1×6 connector and the bridge FPGA, right 1/2 of the PCB is the IC specific to the module and user connector for that IC.

mesa_gpio_cc

Mesa-ICT : Pogo-pin board for programming FPGA EEPROMs post assembly. 1.2″ x 1″ (30x26mm). The header on the right mates either to the Lattice HW-USBN-2B programmer or the Mesa-Logic module for rapid “self-hosting” EEPROM programming post assembly as a bed-of-nails test fixture. Eight 0.050″ pogo pins hold off FPGA configuration and provide full access to EEPROM for external programming.

mesa_ict_cc

Mesa-Backplane : 4 slot back plane that may be cascaded for more slots. 1.3″x0.7″ (33x18mm). Mesa-Bus connections are point-2-point ( WRo to WRi and RDo to RDi ), meaning that cascade depth is limited by the 5V supply current, not the number of loads on the serial bus. Multiple backplanes connect end to end to support 4,8,12, etc modules from a single master.

mesa_backplane_cc

The 5 boards have been sent out to fab and should arrive around mid September 2015. The ICE5LP4K FPGAs in the 2-layer friendly SG48 QFN 0.5mm pitch package are newly released from Lattice and a reel of 50 have been special ordered from Digikey that are due to arrive 1st week of October. Plan is to hopefully acquire some more ES GPUs from FTDI and begin assembling 3 boards of each design to send out for evaluation by interested parties for potential volume manufacturing.

[ Mesa-Modules ]

All Mesa-Modules will be fixed 1″ wide and length available to grow as needed for each design. Most Mesa-Modules will be 1″x1″ with a BOM cost of under $10 for things like GPIO, H-Bridge, Stepper Motor Controller, RTC, thermal sensor, etc . Converting an existing breakout board design from places like Adafruit and Sparkfun to a Mesa-Module should be easy and only be a $5 adder for the Mesa-Bus bridge FPGA and some BML engineering time.

mesa_video

The Mesa-Video design with 3 ICs ( not counting LDOs ) should have a total BOM cost at around $20-$30. The retail cost would need to stay below $50, which should be possible. Mesa-Bus is a solid and easy to use solution for expanding Arduinos with USB like simplicity. Please reply to this post and write to me what you think and what kind of Mesa-Modules you would like to see.

faq

[ Mesa-Video FAQ ]Intel_P8051

o Does Mesa-Video work with devices other than Arduinos? Yes, it will work with any device with a 3.3V LVCMOS ( or level shifted to ) UART serial port.  Got a Teensy, Intel Edison, or even an old 8051 design that you would like to add video to? Bring it on. The bus master sets the bit rate for all devices. This hasn’t been tried yet, but the Mesa-Bus protocol is asynchronous and clear-text, designed to be portable so it should work great over a wireless Bluetooth serial link like a BlueSmirf. With a wireless link it should be possible to use Mesa-Video as a wireless remote display from a desktop PC or Arduino to a TV ( think home security monitoring, stock tickers, etc ), or place the Arduino in a quad-copter and have it stream live video over bluetooth (or other wireless standard) to a remote Mesa-Video display. Clear text protocols like Mesa-Bus are very portable. Designed around old school ( and versatile ) UART serial communications of a wide baud range, it is very simple to convert from LVCMOS 3.3V serial to RS-232 (single-ended) or RS-422 ( differential ) for long runs of to 1,500 meters ( almost a mile for Luddites like me ). SPI and I2C just aren’t designed for this.

Warcraft-3-Difficulty-Selection

o How difficult is the software to integrate into my design? The complexity of the software depends on how many features you intend to use. BlackMesaLabs provides the basic Level-1 ( Text Only ) and Level-2 ( Text + Limited Graphics ) software in both Arduino C++ Sketch and Python for PC examples. Level-3 software is the full up FTDI development kit available here and makes all of powerful FT813 features available. If all you want is to display some debug text information, my screen shot of the Arduino-IDE is all the software that is required.

o What are the Level-1 text modes? When the display is initialized, 1of3 VGA text display modes may be selected. 1x Zoom for 98×36, 2x Zoom for 49×18 or 4x Zoom for 24×9. The graphic text provided by the FT813 can mix and max varying font sizes.

o What are the Level-1 functions? video_setup(), clear_screen(), clear_console(), print_at() and print_ln(). The function print_at() displays a text string at specified x,y location on the screen. The function print_ln() appends and scrolls a text string on a virtual console.

o What will the Level-2 functions be? Most likely draw_text(), draw_point(), draw_line(), draw_rectangle().

oshw-logo-800-px

o If Mesa-Bus and Mesa-Video are open-source software and hardware, where are all the design files? All of the design files are safely stored on the Black Mesa NAS. Once a relationship between BML and a board assembler and distributor is established, the appropriate time and location to upload all design files to the internet ( most likely GitHub ) will be determined.

2141Fig02

o What is the maximum bit rate of Mesa-Bus? It has been tested at 3Mbps, which is the maximum UART rate of the Arduino M0 hardware. Final FPGA bridge design will be clocked at 48 MHz, which assuming 8x asynchronous serial oversampling, a baud rate of 6Mbps should be possible. FPGA to FPGA serial communication where both FPGAs have the same 48 MHz base clock could potentially work at 4x oversampling, or 12 Mbps. One Mesa-Module idea BML is considering ( which may seem ludicrous on the surface ) is a bitstream for the Mesa-Logic board called Mesa-UART, a FPGA design that would take SPI from a CPU ( at say 20 Mbps ) in SPI binary format and stuff the traffic into a rate converting FIFO which is then serialized using a fast UART for Mesa-Bus mastering at 12 Mbps. This may seem like a lot of work ( and overhead ), but for uCs with slow UART serial ports – this is a viable solution for fast data transfers. Mesa-Logic is very flexible and fully open-source FPGA design meaning that a user specific project could easily be adapted requiring only a new FPGA bit stream ( for example, an 8bit parallel interface with IO strobes from an Apple ][ – level shifted from 5V to 3V of course ).

spi_interface_1

o May SPI be used instead of Mesa-Bus UART serial protocol with Mesa-Video? Yes, on the final design the bridge FPGA will have a bypass pin that may be tied to GND. This will bypass the FPGA’s UART and Mesa-Bus decoder and pass SPI signals from the FTDI 6 pin connector to the SPI interface of the FT813. In this configuration, the SPI setup and hold timing will vary slightly from the FT813 spec and the FTDI 1×6 connector will no longer be compatible with a FTDI cable or a Mesa-Backplane. This SPI bypass option is an excellent development interface for someone ultimately targeting a FTDI Flat Panel Module ( See Below ) for a commercial application:

VM800p_new

o Why Buy Mesa-Video when a 512×512 FTDI 5″ Flat Panel is available for $100? Primary difference is that Mesa-Video drives large displays at 800×600 resolution and simplifies digital video from any Arduino sketch without requiring importing a mountain of libraries and understanding the GPUs programming interface. The EVE FT800 series of chips are very powerful – which also means complicated – even just for basic things. Mesa-Video makes the simple things simple without giving up the full power of the FT813. The full FTDI EVE API is still available to use with Mesa-Video hardware.

Atari_Pong_and_Computer_Space

o Will a version of Mesa-Video that supports gaming be made available? Excellent question, this FT813 design actually canceled the original BML GPU design called Video-Duino as the FT813 is much smaller and more cost effective than the Spartan6 FPGA based GPU. The FT813 is a fantastic, small and low cost solution for displaying text and primarily static graphics. If there is enough interest, the FPGA GPU design may be restarted using a Lattice ICE5LP4K  FPGA ( low cost and small like the FT813 ). This would either be designed like a 3DFX like graphics accelerator , a GPU that goes between the FT813 GPU and HDMI chip and overlays real time gaming graphics, or possibly standalone FPGA GPU.  Idea is for the design to provide 8×8 Bitmap Textures and Sprites with velocity and collision detection integrated into the FPGA hardware design ( offloading Arduino of real time work ). It would be a fun development platform for 8bit NES like 2D video gaming. Color resolution may need to be reduced from 24bit to 9bit for a small low cost FPGA to handle the interface. Please respond if there is any interest out there for a gaming specific module like this.

gsm-shield-3

o Is this an Arduino shield ( shown above )? No, but it could potentially replace the shield concept as a much simpler, smaller and more robust method for adding multiple peripheral devices to an Arduino. A Mesa-Bus back plane is in the queue that supports 4 slots ( and can cascade to 8,12, etc slots ) and there is also an Arduino Zero compatible Mesa-Zero module. There are multiple low cost Mesa-Modules envisioned, Mesa-Video just happens to be the first.

o Do multiple Mesa-Modules have to be daisy-chained? No, if you are concerned about latency or series circuit aspect – the CPU master may use a separate serial port pair for each Mesa-Module. Arduino’s Software-Serial makes this very easy.

Create-a-Battery-from-a-Lemon-Step-5

o Is Mesa-Video Low Power? Yes! No promises of powering Mesa-Video  from a single lemon, but using this handy USB Current Meter that measures 5V USB Current in 10mA units, on powerup, with HDMI disabled, it measures current at 0 (so less than 10mA ) and with everything enabled generating video at 800×600 it measures 40mA ( so less than 50mA – Take that nVidia! ). Power-on inrush current has not yet been measured yet. USB 2.0 is rated for 500mA, so a person could theoretically drive 10 HDMI video displays from a single PC over a FTDI cable using 10 Mesa-Video boards daisy chained on a Mesa-Backplane ( think 60″ LCD TVs in a giant 3×3 15’x15′ array…. ). The 0.100″ +5V pin on a 1×6 header is physically rated for 3 Amperes, so the Mesa-Bus has room to grow.

vga_regions_1

o What is the Mesa-Video resolution and timing and may it be changed? The open-source software for Mesa-Video automatically configures the hardware for 800×600 56 Hz timing per VESA Spec VG900601.  This was selected as the FT813 PLL is able to exactly generate the 36 MHz dot clock for 800×600 and 36 MHz exceeds the 25.175 MHz DVI minimum. The FT813 chip itself may be configured for other frequencies, but this falls outside the scope of Mesa-Video hardware+software. Plug fest tests thus far have shown that computer displays with DVI or HDMI inputs agree and work well with 800×600 input feed regardless of their native resolution and aspect ratio.

clip_image002[20]

o What else is Mesa-Bus good for besides Arduino expansion? Remember the days of the LPT1 parallel port when it was easy to add homegrown bread boarded hardware to a PC? A Mesa-GPIO module can bring those days back. A $20 FTDI cable plus a similarly low cost Mesa-GPIO board with a PCA9539 provides 16 5V tolerant IO pins the PC can interface using any language that can talk to a standard COM serial Port ( Perl, Python, Powershell, etc ). Need more than 16 IOs? Buy a Mesa-Backplane and populate it with 2 or more Mesa-GPIO modules.

ftdi

o What other Mesa-Modules from BML are planned? Here is the complete list to date:

  • Mesa-Video : 800×600 HDMI/DVI Digital Video.
  • Mesa-Backplane : 4 slots for Mesa-Modules. Cascadable.
  • Mesa-GPIO : PCA9539APW based 16 3V 25mA GPIO pins ( 5V Tolerant ).
  • Mesa-Analog : AD5592 8 Channel 12bit ADC/DAC/GPIO.
  • Mesa-Stepper : L6470H based Stepper Motor Controller.
  • Mesa-Relay : ULN2803A Darlington based 500mA inductive load driver.
  • Mesa-Logic : Lattice ICE5LP4K based 4K LUT FPGA fabric with user IO.
  • Mesa-DeadBug : Plugs into Mesa-Logic for dead-bugging any IC to 16 IOs.
  • Mesa-Memory : 8Mb Flash, 1Mbit SRAM and 16Kb FRAM ( non-volatile ).
  • Mesa-Environment : Temp, Light, Accelerometer, Altimeter, Magnetometer.
  • Mesa-Time : RTC clock and programmable timers.
  • Mesa-Touch : 11 pin capacitive touch sensor ( AT42QT1111 ).
  • Mesa-UART : 6 Mbps SPI to UART bridge for CPUs with slow UARTs.
  • Mesa-Duino : Arduino M0 derived bus master ( 48 MHz 32bit ARM CPU ).
  • Mesa-ProMini : Arduino ProMini bus master ( 8 MHz 8bit AVR CPU ).

IMG_1193

Please reply if you know which Mesa-Modules you find most interesting or if you think of something missing. My direct email is available at the very end of the BML Welcome Page. The BML goal is to reuse the same FPGA bridge design and crank out multiple modules.  FPGA reuse greatly simplifies design effort, production assembly, test and firmware programming. Goal is for the Mesa-Modules to retail in the $20 – $40 range, just slightly more expensive than a basic SPI or I2C breakout board. The two CPU boards aren’t required of course ( any board with a serial port will work ), but will be much smaller ( 1″x1″ ) and less expensive than conventional Arduino boards as they won’t have IO ( only the 6 pin Mesa-Bus ). The picture above is the Mesa-Zero proto ( final version won’t have the 2×8 connector on right ).

13040-02

Something very similar to this Intel Edison UART adapter ( shown above ) with RX and TX swapped would make for a great Mesa bus master too. Edison isn’t really my bailiwick, but would be a great solution for a project needing Wifi or Bluetooth built in. The clear text UART serial protocol of Mesa-Bus makes it simple to switch CPU architectures.

2266-00_jpg

Electrically, RaspPi would work too – Mesa-Video might not make much sense, but the other Mesa-Modules can dramatically increase ( and isolate ) the IO capabilities of a standard RaspPi.VintageRadioShack_Storefront

o Where can Mesa-Video be purchased? Excellent question.  Sadly, probably not at your local Radio-Shack. Goal is for Mesa-Video to be  a completely open-source hardware and software ( like Arduino ), low cost and readily available in volume for all Arduino developers yearning for a video display. Mesa-Video ( along with other Mesa-Modules ) can greatly accelerate the Maker-Movement by making it much easier for artists to implement their Arduino based designs. BML only does initial prototype assembly and is therefore  actively looking for a production manufacturer and distributor such as Sparkfun , DangerousPrototypes, SeeedStudio or Adafruit that might be interested in Mesa-Video and other planned family of Mesa-Modules. The FT813 GPU chip is not available for purchase just yet, so this design is still in pre-production evaluation until October-2015. Please provide feedback and express any interest in a Mesa-Video board and the Mesa-Module family. Black Mesa Labs aims to assist the Maker Movement by Delivering the Future … 2 Layers at a time.

theend

Mesa-Video : 800×600 Digital video for Arduinos over 2-wire serial Mesa-Bus

“BML Inverted Solder Ball Reflow” process

01_16_2016 Update:

28-LGA.jpg

BML has reflowed 1/2 a dozen of the original QFN packages for the Lattice ICE5LP4K-SG48 FPGA using a hot plate. The same process has recently been modified for use with a LGA-28 packages as well ( shown above – which have no visible pads once placed on the board ). This enhanced process is more reliable and requires less manual soldering ( ideally none ) after reflow. The original method would sometimes result in a misaligned IC due to flux bubbling and lifting the light 7x7mm IC off the PCB prior to solder reflowing. This new process tacks the IC in place with solder – preventing the issues with flux bubbling and the IC moving.

Improved 2016 BML Inverted Solder Ball Reflow process for QFN and LGA devices:

  • Step-1 : Solder bump the PCB pads only for components to be reflowed.
  • Step-2 : Solder bump the QFN / LGA pads of the IC as well ( apply flux too ).
  • Step-3 : Align components under microscope and tack into place using flux paste.
  • Step-4 : Using soldering iron, heat the corner PCB pads going to IC.
  • Step-5 : Solder the ground lug from behind ( if device has one ).
  • Step-6 : Reflow PCB for 4 1/2 minutes on a hot plate ( Aroma ) .
  • Step-7 : Visually inspect and hand touchup pads using soldering iron.

07_24_201532-VFQFN Exposed Pad

IMG_1118

Black Mesa Labs recently developed a unique and new solder reflow process for soldering 0.5mm pitched QFN packages. The process is called “BML Inverted Solder Ball Reflow”.  Rather than apply paste or individually solder bumping the QFN package ( time consuming and expensive ),  the PCB is instead solder bumped ( quick and essentially free ) using a regular soldering iron and standard rosin core solder. Once the PCB is bumped, the QFN is simply reflowed to the PCB using a hotplate. Proper QFN component alignment prior to reflow is crucial and is made possible using flux paste.

The Black Mesa Lab Inverted Solder Ball Reflow Process:

  •   Step-1 : Solder Bump the PCB pads only for components to be reflowed.
  •   Step-2 :  Align components for reflow and tack into place using flux paste.
  •   Step-3 : Reflow PCB for 5 minutes on a hot plate.
  •   Step-4 : Visual inspection and hand touchup QFN pads using soldering iron.
  •   Step-5 : Hand solder remaining components.

How I got on this path is the need to reflow QFN ( leadless ) packages. Xilinx – my goto FPGA provider has gone completely BGA with all their new devices after Spartan6 (45nm) generation.  BGA devices are great unless you are designing 2-layer and 4-layer PCBs and wanting to reflow at home. Basically, I needed a new FPGA vendor that is more aligned with the Maker Movement – small, inexpensive and easy to assemble FPGAs for 2-layer PCBs. Atmel is an awesome company ( they still sell AVR / Arduino CPUs in DIP packages ! ) profiting immensely from Maker community ( all the Arduinos out there ) – but sadly have no FPGA technology.  I discovered Lattice has a 40nm FPGA family called ICE40 that are small inexpensive FPGAs in QFN32 5x5mm and QFN48 7x7mm packages ( majority are still in BGAs of course ). For $2 – $5 I can buy 400 to 5,000 Logic Cell FPGAs that can be soldered to 2-layer PCBs . Compare this to Xilinx Spartan6 XC6SLX9-TQFP144 at 20x20mm at $15 and you can understand my reasoning to chose Lattice over Xilinx for my Maker designs going forward. The picture below shows my new Lattice design ( all components on top side of PCB, FPGA in center ) versus my last Spartan6 Xilinx design ( top side is nothing but FPGA, bottom side has all the other components ).

Lattice Nano-FPGA versus Xilinx Nano6-FPGA Board:

IMG_1126

Back of the larger Nano6 ( with more IO brought out ). Shows all the components on the back.

IMG_0740

QFNs vs QFPs :

48-VQFNsthdmi002abtr

Regarding QFNs ( leadless ) – they are fantastic small and inexpensive packages with great thermal characteristics and compatible with 2-layer PCBs, but difficult to hand solder compared to QFPs ( leaded ) as there are no leads.  They really want to be reflowed in an oven with paste much like a BGA. I almost purchased a toaster oven for BML but Nathan Seidle at Sparkfun.com wrote this great tutorial on using an electric skillet for reflowing surface mount PCBs instead of an oven.  I have decided to try this path for my own prototyping for a number of reasons. One limitation of the skillet method is only populating one side of PCB with components.  If you have seen my previous designs, you know I like to get 2 square inches of components on a 1 square inch PCB – so this is a definite drawback for me.  On the flip-side ( or NOT as it turns out ) I am wanting to start some designs I could potentially have manufactured and sold through Sparkfun – so going single sided for component placement makes sense. Here is a picture of a single sided Sparkfun PCB getting reflowed:

Hot-Plate-Reflow-1

My new PCB designs – “Mesa Zero” and “EVEy Video” are actually my 3rd assembly using the skillet/hot plate method.  The 1st 2 attempts were small QFN32 Lattice ICE40 FPGA boards. 1st attempt was using actual solder paste.  The paste was very messy to apply and post reflow left microscopic solder balls on the PCB solder mask I needed to carefully inspect and cleanup. The biggest downside of solder paste however is the shelf life of only a couple of weeks ( it dries up ). 2nd attempt was using my new Inverted Solder Ball method, but trying to reflow everything with no hand soldering. The small 0603 components tended to bounce around as the flux paste turned to liquid and started bubbling. Both attempts ~worked~, but required a lot of cleanup and hand soldering cleanup post reflow.

One tricky thing about QFNs ( also known as MLFs ) is the substrate ground slug in the center. For many devices – it is the only ground connection and must be soldered – so I place a 0.125″ hole underneath the slug. Unfortunately, this means no routing underneath the package.  For my Xilinx TQFP designs, I always have top-side 1.2V and 3.3V power “planes” underneath the package ( virtual 4-layer ). For QFNs, this isn’t possible as top-side is a ground slug ( and ground is on the bottom side ).

The most difficult thing about QFNs are the leads – they don’t exist – so you can’t see them from above. To check alignment, I need to angle the PCB from all 4 sides.  My 1st attempts I used Elmer’s Glue to hold the package in place while I checked alignment under a microscope.  This actually worked really well, but then I discovered flux paste, solder flux, but in a paste.  A single product replaced liquid flux and the glue. Now I use the flux paste to hold the QFNs in place while I angle the PCB to check aligment. The paste flux melts prior to the solder and becomes a necessary wetting agent ( surfactant ) to improve the soldering process to the pads. Amazing stuff that works great!

IMG_1122

Here is my new procedure in detail, it involves 5 steps but achieves very good results:

Step-1 ) Decide which components I need to reflow on the hot-plate and using a regular soldering iron, solder and liquid flux to tin the pads for those components. This is my “Bumping” process.

OSH-Park Rendering:

mesa_zero

PCB as delivered from OSH-Park:

IMG_0062

Solder placed on QFN and QFPs pads:

IMG_0064


These pictures are the 2 PCBs before and after I solder the pads. The “Mesa Zero” board is a 32bit ARM Cortex M0+  Arduino Zero clone.  The SAMD21 CPU is only $6 and is equivalent in horse power to a Intel 386 CPU from the 1990s containing 256KBytes of flash and 32K Bytes of SRAM in a 7x7mm TQFP48 package.  I designed the board thinking it had a ground slug, but the device does not – so I did not solder underneath that package. The micro-USB connector has caused me grief soldering by hand in the past as the solder tends to flow inside the connector and obstructs cables from plugging in.

OSH-Park Rendering:

eve

PCB as delivered from OSH-Park:
IMG_0066Solder placed on QFN and QFPs pads:

IMG_0067

The “EVEy Video” board is designed around EVE embedded video with a FTDI FT813 800×600 SVGA GPU in a QFN56 package and a TFP410 in a TQFP64 package for reflow and also a small 5x3mm oscillator. This step-1 of soldering the PCB pads with regular solder is equivalent in function to applying solder paste in a normal production assembly.

Step-2 ) After solder is applied to pads I place the components on the PCB and use this Rosin Paste Flux to keep them from sliding around.  The QFPs aren’t that big of a problem as from a vertical viewpoint, I can tell if the 0.5mm pitch leads are aligned.  The QFNs however require me to turn the PCB at 45 degrees for all 4 sides to see if the leads are aligned and is tricky under a 10x microscope. This flux paste keeps the components from sliding around. One caveat is that the paste quickly turns to liquid above about +85F, so the PCB and room must be below that melting point. This picture shows the paste in four corners of each device ready for reflow:

IMG_1100

Step-3 ) Time for reflow.  For less than $20 I bought this awesome Aroma Hot Plate from Amazon

IMG_1107

61O7sGxbKgL._SL1200_

The solder flux seeps through all the vias and makes a sticky mess of things, so I use a sheet of aluminum foil between my PCBs and the hot plate. My 1st attempt I used the “High” setting and I swear the FR4 almost caught on fire ( certainly discolored the OSH-Park Purple to Brownish ). This go around I used “Medium” and it reflowed from “Off” to “Medium” in about 5 minutes. I actually use the hotplate as my permanent “work area” base for microscope work now. Unfortunately, it takes a good 30 minutes to cool down completely.

Step-4 ) Inspection and touchup:

The TQFP had a few leads needing touchup when I inspected with my dental pick.

IMG_1111

The QFN stayed completely aligned thankfully.  I believe that QFNs reflow much like BGAs and tend to “self align” once the solder has turned molten.

IMG_1112

Here is a link to a great video of a BGA package reflowing on top of solder paste slightly out of alignment and snapping perfectly into place. The entire package floats on these molten spheres of solder and aligns automatically.  I believe I am achieving the same results with my QFNs.

Oscillator did well under reflow and stayed aligned.

IMG_1113

I had to remove a bunch of flux ( using a wooden toothpick ) to fully inspect all of the pads on the QFN.  The QFN I could see some “non-silver” on the sides so I went ahead and soldered by hand all the leads just to be certain. The real leads are underneath and can not be visually inspected, but seeing solder on the sides is assurance there is some connection. Once the package is aligned, hand soldering the side “walls” is easy, it is just a matter of rolling a giant solder ball down each side. Step-4 is about soldering all the leads by hand. Step-4 is only possible if Step-3 succeeds in holding the components in place perfectly aligned – which it did. Note that I use really large 0.25 x 1.75mm pads for the QFN when 0.25×0.50mm would normally work.  This makes for easier cleanup as it provides a large surface area for the excess solder to adhere to. Step-4 cleanup is as easy as dumping flux to an area and running my soldering iron tip across it. The solder magically seeks out and adheres to all non solder masked surfaces. No solder wick required. Flux is great stuff. Here is a Before and After of Step-4 on the QFN:

Before Cleanup Pass:

IMG_1117

After Cleanup Pass:

IMG_1118

After Step-4, nothing but silver.  Step-4 is all about hand soldering and inspection of the difficult components. It is imperative to ensure all the difficult components are properly soldered prior to installing the other components as access to the QFNs and QFPs with an iron becomes much more difficult when all the bypass caps, etc are in place. Previously I would use a lot of solder wick for cleanup, now I tend to just use liquid flux as the excess solder will scramble and adhere to the soldering iron tip so long as the tip is clean. This method requires more cleanup ( alcohol and Q-tips – but they are cheap at 1,000 for $10 ).

Step-5, the final step is soldering all of the small components, 0603 and SOT-23, SOIC8 devices and large connectors by hand. Once they are on and I do a final visual component inspection and alcohol scrub and then solder the large through hole connectors. Presenting the final assembly!

IMG_1123

07_25_2015

Spent some time writing FPGA firmware and trying to bringup EVEyVideo. No valid video out yet. Able to toggle the FT813’s GPIO PowerDown output pin, so basic SPI communication is working. Video timing is complicated and this chip has a million registers and proper sequences to follow, so need to persist at it.

07_26_2015

EVEyVideo is Alive!

This morning I succeeded in getting a valid HDMI video signal at 800×600 @ 30 MHz dot clock out of EVEyVideo design. The “@ A B C” are the 1st things I got the GPU to display ( graphics come next ). To save engineering time, rather than try and bring up the FT813 GPU from the SPI Master interface on an Arduino ( and suffer through multiple compile and firmware loading ), I instead created a Nano3 ( Spartan3 XC3S200A ) FPGA design that provides 16 GPIO pins on the Nano Bus header.  This FPGA design can be re-used for anything as all 16 pins are configurable for Input,Output, Drive-1-Only or Drive-0-Only. Having a software bit-bang interface at 1st means I can script everything and iterate rapidly. Here is my 1st image:IMG_1129

Note: About the Nano Bus.  The “Nano Bus” isn’t actually a bus, but an electrical interconnect standard I established at BML for all of my FPGA and peripheral designs.  If you are familiar with PMOD, it is similar, but I think better of course. “Nano Bus” provides 2 GND pins, 5V Unregulated Raw Power ( current limited at 3A by pin ) and 3V regulated ( limited current by masters LDO ). The connectors are color coded Yellow=5V, Red=3V and Green = GND. I can quickly spot 2 PCBs with the Yellow and Green identifiers and know that I can plug them in without blowing anything up. The remaining pins, typically 4,8 or 16 in a dual row configuration are digital IO pins at 3V.  I have splitters that convert a x16 master to two x8s, or a x8 and two x4s, etc.  Also have “Gender Benders” so that I can plug 2 Masters together ( an FPGA and an Arduino for example ). It is a cost effective and versatile interface as I can build a FPGA with a x16 ( 2×10 0.100″ ) Nano Bus Header and plug a small x4 SPI peripheral board into it that is only 1/2″ wide. OSH-Park charges by the square inch, so I can build Nano Bus SPI chip peripherals boards for only $2-$3  that have a 2×4 0.100″ connector ( 2 GNDs, 5V,3V and 4 signals ).

Back to EVEyVideo bringup, so with the Nano3 FPGA as SPI Master to the FT813 in place, I then wrote a Python script for bitbanging the SPI Bits over a standard FTDI USB serial cable. Every SPI clock cycle takes two FTDI serial port writes at 921,600 baud which is REALLY slow of course ( takes about 1 minute for FT813 bringup ). Using this flow means I avoided compiling Arduino C code and flashing firmware. Scripting languages like Python, Perl and Powershell are a tremendous time saver for initial hardware bringup. C is king for final performance but awful for iterating and trying things.

That ends this blog, the EVEyVideo board is fully assembled and electrically all tested out. I will start a new blog in the future specific to EVEyVideo programming and where I plan to take this OSH project. My end goal is to provide a low cost and easy to use video interface for Arduino developers.

“BML Inverted Solder Ball Reflow” process