Lessons Learned: Dealing with COTS

When creating an instrument, we almost always have to deal with some off-the-shelve components (abbreviated here COTS). Very rare are the products that are built almost exclusively out of custom parts! Think of cameras, lenses, mechanical mounts, computers (to process data), cables etc. These are all components that you are not making yourself directly and for which you must rely on a manufacturer and one or more suppliers.

Dealing with COTS is somewhat tricky in a project because of their evolutive aspect: manufacturer can go out of business, suppliers may suddenly increase their price or lead time without further notice, the product itself can change even if the manufacturer claims it’s the same (different housing, materials, built quality…). This aspect of COTS can literally drive managers crazy as they are trying to standardize their manufacturing process while the ground seems to be moving like quick sands! Some manufacturers and some products are more stable than others but on a relatively long period, and on a relatively large set of components, you are likely to see changes occurring during your product’s lifetime.

By the way, denial, while initially tempting, is in no way a solution. Trying to push the responsibility to the manufacturer or the supplier is only an illusion. Maybe you can obtain some additional stability if you are a big contractor, but who is big enough to tell the camera or IT market what to do? Do you really believe you can get Samsung to have your own production line of ceramic capacitors for the next 20 years to come? Even smaller manufacturers like Thorlabs, Edmund Optics, Newport etc. won’t listen to your complaints. The best you can expect is to receive an email telling you something like “we’re deprecating element XYZ – 200 pcs are still available in stock – first arrived first served”. I’ve seen this pattern repeat again and again; even as quick as 6 months after the supplier guaranteed you that the part would be available for at least 5 more years! True story.

So, if denial is not an option, what can we do to mitigate the problem? As always, start by embracing change. If your processes can’t cope with change, they are bad processes. Change them, now.

Let’s have a look at the conventional way to build a product – or, at least, what I was taught was the conventional way. Then, we’ll improve on that with a few suggestions.

A typical process is summarized in Figure 1. We start by drawing out some specifications for the product, then hand on to design where all the CAD and engineering comes in (eventually split into a preliminary and definitive design phases separated by milestones), orders are placed for all the parts once all the drawing are ready, product is assembled and finally tested to confirm that it’s meeting the specifications. During production, only the purchase & integration phase is kept since the development is finished and it’s out of question to change what the product is doing (specifications) or how it’s doing it (design). Some simplified testing is also performed to check that the product behaves as intended (quality check / acceptance testing).

While there is a lot to say about the process of Figure 1 and why it’s so, so, wrong, I’ll focus on the purchase & integration part for today. Despite the chart does not give any direction on how to place the orders, I’ve noticed that in the development stage, since the design has been frozen, parts with the longest lead time are often ordered first and the part with faster deliveries are ordered later. This is somehow logical (or at least looks like it is) in an environment where people are always late and busy on other projects as you are taught to do the most urging thing first. Typical lead times are on the order of up 3 months for custom lenses, 6 weeks for custom mechanical parts and 1 week or less for COTS with some notable exceptions (hand-built parts or parts experiencing shortage issues). It’s therefore common to rush on the orders for custom parts and place them as soon as the drawings are ready & validated and wait until the last minute or so to order the COTS. In a production environment, where you have everything set up in your ERP, all purchase orders are usually generated at once. The important thing to remember is that the usual process does not do any ordering in the purchases. And that’s a hell of a problem as I will show now.

I will illustrate a common issue met with the process of Figure 1 with the camera sub-assembly of the OpenRAMAN spectrometer shown in Figure 2. It consists of a custom lens bracket, a COTS machine vision lens, some screws, and a camera.

The sub-assembly was made according to the traditional workflow that I presented here-above. At first, the system was designed such as to evaluate total cost. The budget was then discussed in the NPO for official approval, and, once approved, all parts were ordered. So far, so good.

The problem occurred when all the parts came in: the machine vision lens was not fitting in the bracket. After a few minutes of investigation, it came out that the lens housing diameter did not correspond to what was advertised in both the STEP file and the mechanical drawing. And it’s not that there was just a problem with the STEP and drawing because it was a time where the camera lens had these dimensions. However, the manufacturer updated the design at some point between 2019 and 2023 and did not reflect the change in the public documents. The part is then a different one although it is sold under the same part name/id. And we’re now stuck with a part that does not fit into the specially designed bracket.

The only solution at this stage is to rework the bracket or order a new part – and ordering a new part sets you back by about 6 weeks, not to mention the cost of the new part itself. And this can happen during the development phase and in the production phase as well. Imagine the development in Figure 2 went successfully and 6 months later I purchase 100 of these to discover that all the parts are now wrong!

Now that the problem is clear, what can we do to limit the risk of wasting both time and money? Well, and why wouldn’t we buy the COTS first and not the longest lead items? That is going to be my provocative proposition for this post.

By purchasing the COTS first and check all the important properties before validating the purchases of the long lead items. If something goes wrong, we can now correct the custom parts before they are ordered – avoiding therefore the need of a rework or repurchase. The only thing this strategy cost us is a bit of time: (1) time of delivery of the COTS, and, (2) time of inspection. I’m not counting in the time for eventual modifications of the CAD because we would have to do it anyway if parts wouldn’t be fitting. For most COTS suppliers, this would set you back by only a few days on the overall delivery pipeline but it saves you from the unpleasant surprise of delays of several months. Unless you like gambling in a casino economy, this is a very rational way of doing and I don’t think there is a lot to argue about it.

In that methodology, there is an inspection part required to validate that the COTS are okay. During that phase, we will be checking the interfaces between the COTS and the custom parts. For the clamp of Figure 2, that would simply means measuring the housing diameter of the objective to be sure it matches the one on the CAD. You could also check that the camera is indeed a “C” mount camera and not a “CS” mount as you wouldn’t be able to adjust focus properly (CS mount is shorter than C mount). It’s very unlikely that the camera manufacturer will switch from C to CS mount (not to say impossible) but it’s always difficult to know in advance where the problem comes from. What could have happened, for instance, is that the mechanical designer would have used a generic STEP file of a C-mount camera provided by the camera manufacturer while the camera is actually a CS-mount. Things like this happen in the real world, sadly.

Also don’t limit yourself to big or expensive components, screws and pins can be wrong too! That’s something that I encountered a lot: DIN7 pins ordered with bevels are slightly longer than the unbeveled pins used in the CAD software library. If the pinhole doesn’t account for the extra bevel, you get two mechanical parts that will never get in contact with each other. This is represented in Figure 3.

This may look daunting at first sight, but an experienced eye will make the difference. When you don’t know, check all the faces in contact with custom parts in the CAD. When you’re confident, you can skip some inspections.

And I hear you saying “wait, why skipping inspections if the purpose is to be sure there is no mistake left?”. Well, there are two things I can reply to that.

First, never believe you will not make any mistake at all. The process aims at lowering the risk, not annihilate it. The thing with the pin length is particularly difficult to spot but usually you will make that specific mistake only once. It’s the kind of thing you must go through yourself and that you never do again after. That’s where a trained eye can spot the difference when reviewing other engineer designs. Remember that I have seen the pin problem multiple times in design – just not made by the same people.

Second, it may not be worth checking 100 objectives of Figure 2 if you bought all of them at once and the first one you inspected is alright. If one (or more) of them has a different shape, you’re likely to spot it as you open the box (the human eye is very sensitive at spotting differences between two identical parts next to each other). Micrometer-level differences in housing might not be worth checking as well because your design may accommodate some small fluctuations. For instance, the clamp of Figure 2, can easily accommodate up to ±0.25 mm in housing diameter which is well above typical machining tolerances you can expect from a mass production.

Now, what do we do when we spot a difference? The first reaction shall always be to consider the mistake is on our side (e.g. wrong component selected, wrong reference in purchase order etc.). If the divergence comes from the manufacturer or supplier, you can then either choose to contact the manufacturer/supplier and ask for a replacement or you can decide to change the custom part on your side to avoid longer delays. In some cases, contacting the manufacturer or supplier will not be of any help because they will propose you at best to send back the product you’re not happy with but won’t provide any solution for replacement. And even if they do, you’ll likely have to wait again.

So, I’ll assume you took the decision to change your CAD. Then comes the question: how to version the change?

Imagine the custom part is currently under revision “rev0” and you’re going to implement a modification to accommodate to the received COTS. Should this become revision “rev1” or something different?

It might look tempting to create a new revision of that part with, for example, an updated bore hole diameter (as in Figure 2). But what will you do with the next production order? Will the base design be revision “rev0” or “rev1”?

Since we’re creating a part that does a different function (somehow), my recommendation today would be to create a part variant id at revision “rev0”. So, if the original part designed to accommodate 32 mm objectives was “#2023-15/0” (part #2023-15 revision 0), you would now have a different part “#2023-15/1/0” (part #2022-15 variant 1 revision 0) to accommodate 34 mm objectives. Mistakes or upgrades like chamfers, more fastener holes, would then follow the normal revision pipeline again (revision 1, then 2, 3 etc.). You can adopt the naming convention that “#2023-15/0” means “#2023-15/0/0” (part #2023-15 variant 0 revision 0). In traditional version-control strategies, this would correspond to a simple form of branching (or forking) of the initial part you designed. And because you now have a tree of development, keeping a log file with all the variants of the parts and revision is a good idea.

You may argue that this complexifies the whole purchasing process since with all the parts changes leading to new parts, we don’t know what to purchase at the next production batch. But remember that we’re only adapting custom parts here, not the COTS. So, the BOM (bill of materials) for the COTS doesn’t change unless one of the parts got deprecated and you’re still buying the same COTS. What eventually changes is which variant of the custom parts you’ll be purchasing. And this is precisely where your logging tree comes handy: as you receive and inspect the COTS parts, you can decide which variant you need to order and substitute it during assembly. For this to work, you need to keep the same mechanical interfaces between all your custom mechanical parts. If for some reason this is not possible because you need, for example, to change fastener holes positions, you would need completely new parts or create revisions for all the variants of the parts involved. This should, however, be an extremely rare situation.

We can summarize this as the logical flowchart of Figure 4. Suppose you want to create the assembly of #2023-01/0 which has its own BOM of COTS and custom components. You would start first by purchasing all the COTS of the assembly and sub-assemblies. After reception, you would perform an inspection of all interfaces with custom parts. If for some of the components you see some discrepancy, evaluate if you need to return the part or perform a modification of the design. If you choose to perform a modification of the design, create a variant of the part that needs to be updated and of all the parent assemblies to accommodate for the change. Then place the order of the custom parts and log the complete assembly id, including variant, that was used to produce that batch.

That’s all for today’s grain of salt!

Want to discuss this further? Check out our new [∞] community board!

I would also like to give a big thanks to Young, Naif, Samuel, James, Sebastian, Lilith, Hitesh, Alex, Jesse, Stephen, Jon, Sivaraman, Cory, Karel, Themulticaster, Tayyab, Marcel, Kirk and Dennis who have supported this post through [∞] Patreon. I also take the occasion to invite you to donate through Patreon, even as little as $1. I cannot stress it more, you can really help me to post more content and make more experiments!