Check out this Miscellaneous WhizzML script "Remove Anomalies"

Remove anomalies | BigML.com

FREE

Remove Anomalies

Details:

Created	Thu, 8 Oct 2026 15:04:04 +0000
Published	Thu, 8 Oct 2026 17:25:33 +0000

Description:

Remove the top n anomalies from a dataset or the ones over a maximum score. By default, the top number of anomalies returned in the anomaly detector will be removed. You can choose to limit by number or by max-score. If both are set, both limits will be used.

Tags:

BigMLer BigMLer_ThuOct0826_170404

URL:

https://bigml.com/user/whizzml/gallery/script/6ac7b0e4a5882b68cc0a084a

Script FREE

Source code

;; Remove the top n anomalies from a dataset

(define TOP-N-DFT 10)
(define SEED "bigml")

;; Given an anomaly resource, get the list of its top_anomaly row numbers
(define (anomalous-rows a attr)
  (map (lambda (x) (x attr)) (a ["model" "top_anomalies"])))

;; Given a list of row numbers, generate a flatline expression
;; that discards those rows.
(define (row-filter rows)
  (let (eqs (map (lambda (n) (flatline "(= (row-number) {n})")) rows))
    (flatline "(not (or @{eqs}))")))

;; Given an anomaly and a datset, create a batch-anomaly-score and return
;; the associated dataset
(define (make-b-a-s-ds a-id ds-id)
  (let (b-a-s (create-and-wait-batchanomalyscore {"anomaly" a-id
                                                  "dataset" ds-id
                                                  "all_fields" true
                                                  "output_dataset" true}))
    (wait ((fetch b-a-s) "output_dataset_resource"))))


;; Given a dataset and a number of anomalies, generate a new one
;; that removes from the original the anomalous rows.
(define (normalize-dataset dataset-id n max-score)
  (when (and (= n -1) (= max-score -1))
        (raise "You need to set a number or a maximum score"))
  (let (t-n (if (> n 0)
                n
                TOP-N-DFT)
        a-id (create-and-wait-anomaly {"dataset" dataset-id "top_n" t-n
                                       "seed" SEED "anomaly_seed" SEED})
        anomaly (fetch a-id {"exclude" "trees,fields"})
        anomalies (anomaly ["model" "top_anomalies"])
        scores (filter (lambda (s) (> s max-score))
                       (anomalous-rows anomaly "score"))
        rows (drop (- (count anomalies) (count scores))
                   (anomalous-rows anomaly "row_number"))
        dataset-id (if (and (> max-score -1) (= n -1))
                       (make-b-a-s-ds a-id dataset-id)
                       dataset-id)
        filter (if (> n 0)
                   (row-filter rows)
                   (flatline "(> (f \"score\") {max-score})"))
        ds-args (if (> n 0)
                    {"lisp_filter" filter}
                    {"lisp_filter" filter "excluded_fields" ["score"]}))
     (delete a-id)
     (create-and-wait-dataset (merge {"origin_dataset" dataset-id}
                              ds-args))))

;; Script with parameters dataset-id and n
(define normalized-dataset (normalize-dataset dataset-id n max-score))

Description

Inputs

Outputs

License: Creative Commons Zero V1.0
Script FREE

Remove anomalies | BigML.com

Sending Request...

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY

License

Embed this Script in your web site

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY