Check out this Miscellaneous WhizzML script "Deduplicate"

Deduplicate | BigML.com

FREE

Deduplicate

Details:

Created	Thu, 8 Oct 2026 15:03:55 +0000
Published	Thu, 8 Oct 2026 17:25:38 +0000

Description:

Select one row out of each group of contiguous rows with the same value of a given field

Tags:

BigMLer BigMLer_ThuOct0826_170355

URL:

https://bigml.com/user/whizzml/gallery/script/6ac7b0db426c2387bb0fdd66

Script FREE

Source code

;; A filter that keeps each of the contiguous rows of the given field
;; with probability 1/(remaining window size) each.
(define (reservoir-filter field)
  (flatline "(let (id (f {{field}})"
            "      n (count (cond-window {{field}}"
            "                            (= id (f {{field}})))))"
            "  (= 0 (rand-int n)))"))

;; Filter that keeps only the first of every batch of duplicated
;; fields.
(define (uniqueness-filter field)
  (flatline "(!= (f {{field}}) (f {{field}} -1))"))

(define result
  (let (id (create-dataset dataset-id {"lisp_filter" (reservoir-filter field)})
        id2 (create-dataset id {"lisp_filter" (uniqueness-filter field)}))
    (wait id2)
    (delete id)
    id2))

Description

Inputs

Outputs

License: Creative Commons Zero V1.0
Script FREE

Deduplicate | BigML.com

Sending Request...

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY

License

Embed this Script in your web site

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY