1500 tags destroyed during import process

I have a folder containing 1500 tags or so.

I exported it to csv, adjusted the folder structure and some tag names

re-imported it and got the error ‘null reference at index 7284’ or something to that effect

Opened my csv file up to check the contents. Looked good so I thought I’d re-export on top of it and just do a few tags to see what was going wrong.

I exported again but for some reason only one tag was showing up (a folder tag)

Then I noticed red xs on the tags in my tag tree

Then my ignition designer got super slow. So I thought I would re-open it to speed things up.

When I re-accessed the folder I found that my tag folder had disappeared.

And now I’m praying that there is some way of recovering these tags

Any Ideas on how to do this? My last backup is from a month or so ago and I’ve done lots of changes since then.

First thing I would do would be to go to “{Install Dir}\data\db\autobackup” and copy out whatever is there to a folder somewhere else.

The auto backup system keeps the 5 last system states, created when the internal database changes. It’s possible that you have one from before you made the changes. Look at the timestamps to see.

If you do, you can do the following:

  1. Stop Ignition
  2. One level up, in the “db” folder, copy all of the “settings.*” files into a folder, just to get them out of the way.
  3. Extract the contents of the autobackup to this folder. You should now have the “settings.*” files from that version.
  4. Start up Ignition.

If you don’t have an auto backup from an appropriate time, the next thing to do would be to open up the internal database directly and look at the state of the tag tables. You can do this using the command contained in this batch file. Put the batch file in the “db” folder. Edit it to point to the correct location of hsqldb.jar (now in “{Install dir}/lib/core/gateway”), and with Ignition stopped, run it.

The main tables for the tags are “SQLTAGS” and “SQLTAGPROP”. You can select, delete, etc. as much as would help. Then, when done, run “shutdown” (without quotes) to close the db file.

If you don’t feel up to all this, of course, you can call in tomorrow.

Hope this helps,

Does it matter in what order the tags defined in the csv file are listed?

I am trying to trouble shoot this and am just trying to add one record but I get multiple tags being destroyed.

The process is this. I click the root tag folder, then click export tags.
I then add one additional row to the file in excel, saving back as a csv file. The record is listed below.

ML/ 24HRTOTAL 0 4 FALSE Read_Write IgnitionOPCUA [MAIN150]Global.F162.F162[99] Default 0 0 100 0 10 0 0.0001 #,##0.0 0 100 WEIGHTOMETER TOTALIZER 0 0 1 1 0 0 0 FALSE 0.01 0

I then try and reimport it but a lot of tags stored in other folders (not the ML/) no longer show up (i.e. the whole folder is empty)

Any clue why? I used to do this in 7.2.4 no problem but haven’t been able to add new tags this way since coming to 7.3.1… its been a while so I might be missing something)

Cheers

Alex

Ok I got it to work, but only by deleting any existing tags listed in my tag import (so that only new tags are being imported).

To get the tags where I wanted them I had to import them into a new folder (by creating a new folder, selecting it as the target, and then clicking on the import tags button). I then had to drag them into their respective folders and delete the originals.

Interestingly I got errors trying to cut and paste the tags (approx 900 new tags) and could only get it to work by copying them to the the destination folder and then deleting the originals.

Since I presume that importing existing tags starts some sort of builtin ‘UPDATE’ routine with regards to the tag database, and creating new tags involves an ‘INSERT’ routine. -It seems that the UPDATE routine has some problems with operations involving large amounts of tags and/or folders.

The usual symptoms of this are empty folders after an import tags operation.

I’m hazarding a guess that the cut and paste tag operation is also an ‘UPDATE’ operation and is running across similar problems (its null reference errors that I get when trying to do this).

Completely guessing as to how you guys do things but hopefully I’m not too far off.

-Alex

Here is the error message encountered during importing 4220 existing tags (ie trying to overwrite an existing folder structure) and importing 856 new tags.

com.inductiveautomation.ignition.gateway.sqltags returned null for key 19089

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore$5@b5840 returned null for key 19089. at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:283) at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:253) at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:214) at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:699) at com.inductiveautomation.ignition.client.sqltags.impl.db.DatasourceTagManager.addTags(DatasourceTagManager.java:346) at com.inductiveautomation.ignition.client.sqltags.impl.ClientTagManagerImpl.addTags(ClientTagManagerImpl.java:269) at com.inductiveautomation.ignition.designer.sqltags.action.ImportTagAction$CSVTagImporter.run(ImportTagAction.java:247) at com.inductiveautomation.ignition.client.util.gui.progress.ClientProgressManager$ClientRunningTask.run(ClientProgressManager.java:544) at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:526) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.google.common.collect.NullOutputException: com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore$5@b5840 returned null for key 19089. at com.google.common.collect.MapMaker$StrategyImpl.compute(MapMaker.java:617) at com.google.common.collect.MapMaker$StrategyImpl.compute(MapMaker.java:469) at com.google.common.collect.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2046) at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.getDefinition(InternalTagStore.java:1183) at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.getProperty(InternalTagStore.java:1188) at com.inductiveautomation.ignition.gateway.sqltags.execution.tags.AbstractGatewayTag.getAttribute(AbstractGatewayTag.java:108) at com.inductiveautomation.ignition.gateway.sqltags.providers.ProviderSubscriptionManagerImpl.decrementSCRefCount(ProviderSubscriptionManagerImpl.java:107) at com.inductiveautomation.ignition.gateway.sqltags.providers.ProviderSubscriptionManagerImpl.updateRemoved(ProviderSubscriptionManagerImpl.java:300) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.handleDeletedTags(AbstractStoreBasedTagProvider.java:423) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.tagstoreItemsRemoved(AbstractStoreBasedTagProvider.java:304) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractTagStore.notifyItemsRemoved(AbstractTagStore.java:97) at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.deleteTags(InternalTagStore.java:698) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider$AddTagsTask.run(AbstractStoreBasedTagProvider.java:691) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.runInConfigLock(AbstractStoreBasedTagProvider.java:1074) at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.addTags(AbstractStoreBasedTagProvider.java:454) at com.inductiveautomation.ignition.gateway.sqltags.providers.RedundantProviderWrapper._addTags(RedundantProviderWrapper.java:64) at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.inductiveautomation.ignition.common.messages.ReflectiveInvokeMessage.execute(ReflectiveInvokeMessage.java:41) at com.inductiveautomation.ignition.gateway.sqltags.providers.RedundantProviderWrapper$ConfigRedundancyMessageListener.receiveCall(RedundantProviderWrapper.java:190) at com.inductiveautomation.ignition.gateway.cluster.QueueableMessageReceiver.receiveCall(QueueableMessageReceiver.java:45) at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.dispatchMessage(RedundancyManagerImpl.java:619) at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl$ExecuteTask.run(RedundancyManagerImpl.java:640) ... 9 more

Thanks. I thought we had that problem fixed, but apparently not. We’ll look into it.

Regards,