Preparing your facility for the unthinkable

Back in mid 2007, I was working in my office when when a fire alarm went off.  I knew that this was a serious situation because:

  • The alarm is only triggered by a flowmeter due to a sprinkler valve breaking.
  • Most employees had left for the day, greatly reducing the likelihood of an accidental breakage.

We had our accidental breakages and dealt with them with a sense of humor.  You could always tell whom was directly involved by pointing out the soaked employee in the lineup.  But the smell of burning plastic outside my door alerted to me that this was no drill.

The plant was at zero capacity for one day, and at one-half capacity for 6 days (two weekend days included).  The minimal downtime considering a burned out production line is a great success story towards disaster preparedness.

Tornados, earthquakes or fires can happen at any time.  Two of the three I have experienced in a production environment that resulted in emergency actions and repairs to be taken.

To hell with production, shut down the affected building.

And shut down the entire building before vehicles with red lights show up.  In our case, we had a ‘go kit’ prepared to shut down subsystems in the event of an emergency.  Consisting of a screwdriver, wrench and lockout gear, usually this is intended to shut off equipment that has failed destructively.  In this case, it locked out the building.

First shut off the natural gas or propane

Most equipment with burners will interlock off with a loss of gas pressure.  This removes an energy source that can feed a fire or fuel an explosion.  Make certain that your go-kit includes an adjustable wrench capable of operating this value.  Know the shortest route to this valve which is almost certainly located on the outside of the building.  If you gas-train has multiple valves, your energy supplier will gladly support you in identifying the most effective valve.

If you have equipment that runs on external fuel storage, such as diesel fuel or propane, those also must be disabled by closing the fuel line, or if that is unavailable, locking out the equipment.  This kind of equipment is typically air compressors or backup generators.

Lock out every valve you shut with a real lock.

Shut off the electricity

Mind the backup generators.  Disable those first.  Backup generators don’t apply to these situations; they are designed to replace external utility failures.

Know what you need to do to pull the switchgear.  Maybe even practice once during a time the facility is not operating.  Switchgears have a limited life and can only be pulled 10-15 times before needing service, so use this operation wisely.  They can be difficult to pull as you are powering a spring-loaded clockwork that powers a mechanical current interrupter apparatus once the lever is pulled to a sufficient distance.  Do not bother with the smaller switches first as you are increasing the likelihood of breaking the equipment for no appreciable increase in safety or minimization of liability. It is cheaper to rebuild one switch, instead of two or three.

Lock out every primary switch with a real lock.

If possible, vent the compressed air

This is not such a big deal, but a luxury item.  In the unlikely event of a collapse fracturing a pipe, the air could rapidly oxidize the flame, increasing the efficiency of the combustion.  Vented air is very loud and dangerous, be physically prepared for this.  Projectiles of dust and oil will leave the line as well as lift dust in the area, creating in inhalation hazard.  Protect your ears and quickly vacate the area.

Cap the drains

There is a lot of water, and if your environment includes hazardous chemicals, the fire department water can be interpreted as hazardous waste.  Make a good effort that loading dock drains, sump pumps are not able to release to the city sewer or storm drains.  Sweep the resulting water into the areas.

In our case, a loading dock as an improvised storage pond.  A rush order of 55 gallon drums had the fluids contained.

Reap the benefits of your efforts

Notice I said above to use a real lock.  They are difficult to defeat and are considered a viable lockout mechanism in most scenarios.  It has a secondary benefit.  It often occupies a single lockout point that prevents the utility companies from attaching their own lockouts.  Most of the time, the emergency responders are satisfied seeing a lock, regardless of origin, attached to sources of hazard.

When a city agency or utility places a lockout, you can be forced to undergo a lengthy assessment of the entire building (why you lock out the entire building instead of the affected region).  City inspectors are busy people and there is often a scheduled time a few days in the future for them to inspect repairs.  If you are non-compliant in your repairs, the entire building can remain disabled during the time the next meeting is scheduled.

You can negotiate with the inspector to justify turning on regions of the building that you can prove to be isolated from the utilities by additional lockouts, as was the case with our fire.  We had secondary switches that enabled certain regions and equipment in this building.  We locked out the ones in the affected regions and when the inspector was assured of our configuration, we were cleared for operation.  This all occurred on the morning of the morning after.

Build your own shutdown strategy

  • Know all the paths from your go-kit to each of your energy sources.  Shut off natural gas, then liquid fuel, then electricity, then compressed air.  Do not let your path traverse across production lines so that you don’t run into equipment or become snagged by a hook handing from an overhead line.
  • Test that the tools work at each of their destinations.
  • Lockout at every point.  Make certain that you know how many locks you need.
  • Discretely show the inspector that events occurred to a plan.  They will have a greater confidence in your efforts and will afford you less scrutiny and a higher degree of self-assessment of your efforts
  • Identify the low points where contaminated water will collect.  If there is storage that cannot become wet, determine risks of this flooding.

Better and more meaningful experimentations

When I was an undergraduate at Iowa State in the mid-2000’s, I became fascinated with the research that was conducted under the programs or Vikram Dalal and Gary Tuttle.  My self adopted mentor, Dr. Curtis Sell whom I originally knew through the amateur radio club on campus and also recruited for my tutoring-for-pizza venture, was a very active participant in the microelectronics program.

The joke in that department was that “You get your masters for repairing the equipment and your PhD for using the equipment you repaired”  The equipment consisted of a PDP-11s and other mid-1970’s era equipment that had been likely discarded from prominent semiconductor fab firms that regularly contribute to the college.

Despite the old equipment, much of the work I had witnessed was groundbreaking glimpses into the future.  Photonic bandgap research that gave birth to on-the-die RF channels (waveguides) happened in this lab.  I was fortunate to have surrounded myself with some of the more brilliant graduate students doing dazzling work.

I also got to see the other side.  Those high on the ‘what-on-earth-is-he-trying-to-do’ quotient.  Usually confident, and somewhat self-assured, yet hiding in an office space, strategically arranged so that their desk, monitor or work area is entirely obscured from the vantage point of the doorway window.

It is from one of these people whose experiment I witnessed stuck with me to this day.

Imagine the world of radio.  Terrestrial radio signals almost universally appear deliberately from, or as a consequence of an electronic circuit.  These circuits, and the antennae that radiate signals are almost exclusively made from metals.  Metals allow for a very easy electron flow, so they are ideal conductors, but they also can cast very elaborate shadows (imagining radio waves as visible light) and diffraction patterns.  Modern human beings are well aware of this as we have adapted by learning the ‘cell phone dances’ which include steps such as the spin-around, the stretch-neck-tilt and the circling-wanderer. What you are doing is finding the sweet-spots that exist as columns and rays criss-crossing the area due to diffraction pattern due to building, steel rebar or radio interference.

With this kind of knowledge expected of an ISU Electrical and Computer Engineering Department undergraduate, I was dumbfounded to see an antenna experiment set up incredibly awkward.

Two microwave horn antennas facing each other, the 1/2″ adjustable steel rod that the antenna was attached to extended outwards and in front of the horn.  Observing the optics of the experiment, I was fairly confident that the receiving horn would be seeing diffraction patterns due to the mounting apparatus, patterns that will vary as a function of the changing distances made during the experiment.  When I asked about the validity of the experiment, I was told, “We tried warning him.”

Not a useful RF experiment.
Not a useful RF experiment.

If you are doing an experiment, do not be too confident to ask your peers if they see any flaws that may wreck your experiment.  The Journal of Irreproducible Results exists as a joke, and no, they seldom make calls for papers.  Use skilled and trusted people around you to make your work great, but if even a total goofball makes a suggestion about something being invalid or incorrect, take the criticism with a grateful appreciation and give the comment or idea a thorough analysis.

This semester, I am studying under Dr. Yadav of North Dakota State University in his course of Experimental Design.  The assumptions at this level are that your apparatus is reasonable, but approaches a scientific method in which experiments are conducted.  It is possible to destroy the validity of a perfectly good experiment by using even the wrong techniques during measurement, and your apparatus may be of a highly-elegant configuration.

Randomized ordering of experimental controls and statistical analysis of your data with multiple runs will give you a bank of data where noise can be filtered using statistical models.  It is not good enough the step through your controls in a linear manner, taking repeated data points on each pass, but instead randomize the experiments such that each pass requires changing at least one control.  Do not repeat your data points multiple times with the same control values, instead replicate your data points by performing each experiment as if it were the first pass.  Save the data aggregation for the statistical analysis use the data is collected.