- Release Notes
- Getting started
- Notifications
- Projects
- Datasets
- Data Labeling
- ML packages
- Out of the box packages
- Pipelines
- ML Skills
- ML Logs
- Document UnderstandingTM in AI Center
- AI Center API
- Licensing
- AI Solutions Templates
- How to
- Basic Troubleshooting Guide
- About troubleshooting
- ML Skills
- Pipelines
Pipelines
This section provides frequently encountered errors related to Pipelines.
A pipeline run is failing due to ML Package issue.
A potential cause for this error can be a wrong minor version chosen when running the pipeline.
When deploying a new package, the only minor version available is 0. The reason for this is because there are no pipelines executed on this package yet.
Training pipelines
If you are deploying a training pipeline, we strongly recommend you always use minor version 0.
Full pipelines
If you are deploying a full pipeline, we strongly recommend you always use minor version 0.
Evaluation pipelines
Evaluation pipelines are used to evaluate a trained ML model. You can execute this on any version of the ML Package to get the corresponding evaluation scores. This is similar with grading or evaluating an ML model with the evaluation dataset.
- Pre-trained packages (out of the box packages): since these packages are pre-trained, run the evaluation pipeline on the minor version 0. To evaluate the model after training using specific data, choose the minor version you want to evaluate (trained version).
- Document Understanding:
- Since these are generic, retrainable models, the models need to be trained first. Only run an evaluation pipeline when the models are trained and a new minor version of the package is available.
- Select the most recent minor version, or any other minor version (except 0), for which the evaluation scores can be obtained.
Pipeline is killed automatically
Pipelines are automatically killed after seven days to avoid being stuck for longer periods of time and consuming licenses. Follow the recommendations below.
- Enable GPU.
- Apply dataset optimization technics.
- Reduce the number of EPOCHS.
- Waiting for License
- Running
- Failed
- Killed
Check the sections below for more details on each status.
- Enable GPU
- Optimize the dataset. For detailed information on Document Understanding datasets, check the Training High Performing Models page from the Document Understanding guide.
- Open Automation Cloud™.
- Go to the page.
- Check if the corresponding AI Units are available.
- Select the stuck pipeline.
-
Check the Logs section.
- If the logs are recent and are streaming, the pipeline is in progress.
- If the last log is generated a long time back, download the logs using the Download button and share it with our Support department. If the download button is not visible or disabled, copy the logs from the Logs section and share it with our Support department.
If the pipeline run is in the Failed state, check the possible reasons below.
Check that document type data is in dataset folder and follows folder structure
The following error occurs:
Error: Document type data not valid, check that document type data is in dataset
folder and follows folder structure.
The format of the folder provided for training needs to be in the dataset format.
- Make sure that the provided dataset is correct.
- Make sure that the provided dataset is exported from Document Manager. For more information on datasets related to Document Understanding, check the Export Documents page from the Document Understanding guide.
- In case of scheduled pipelines for automatic retraining loop, select the folder containing the exports from the Data Labeling sessions and latest.txt.
Images or directory does not exist or is empty for invoices dataset
The pipeline run fails because images/directory does not exist/is empty for invoices dataset.
The dataset path provided for either training dataset or evaulation dataset is empty.
To fix this, update the dataset path for evaluation or training, according to the pipeline.
Unschedulable nodes are available
Unschedulable 0/n nodes are available error, contact our
Support department with the your Automation Cloud™ tenant information.
No space left on device
No space left on device error, contact our Support
department with the your Automation Cloud™ tenant information.
The Killed status is usually displayed when the pipeline was killed by the user. For more information on managing pipelines, check the Managing pipelines page.
If the pipeline status is Killed without any user intervention, the most common reason is that pipelines are automatically killed after seven days. For more information on pipeline statuses, check the About pipelines page.
A pipeline run is failing due to dataset structure, input parameters, path, folders, or evaluation dataset.
The following error occurs:
#Error: Training and / or test set is empty, verify that training / test split is correctly set in split.csv
This error is most commonly is usually caused by the wrong dataset format or incorrect ratio of train and validation set in split.csv. Check the Training Dataset page for general guidelines on how to create a training dataset.
The following error occurs:
#Error: Training failed for pipeline type: FULL_TRAINING, error: Full / evaluation pipelines require an evaluation dataset. Please re-run the pipeline providing an evaluation dataset
This error typically occurs when an evaluation dataset was not provided. Check the Training Dataset page for general guidelines on how to create a training dataset.
When running a pipeline, it can occasionally fail because of an Insufficient Licenses error in the Pipeline Data page.
This error does not imply the absence of an actual license. Rather, it signals that all available AI units, whether at the tenant or organization level, have been consumed.
Use this procedure for AI units at organization level.
Use this procedure for AI units at tenant level.
- Pipeline failed due to ML package issue
- Choosing the correct minor version when running pipelines
- Pipeline killed automatically
- Pipeline running for too long
- Waiting for licenses
- Running
- Failed
- Killed
- Pipelines failed due to datasets issues
- Wrong dataset format
- Evaluation set not provided
- Pipeline failed due to insufficient licenses error
