Filter dataset on row population percentage

Filter dataset on row population percentage | BigML.com

FREE

Details:

Created	Thu, 8 Oct 2026 15:03:51 +0000
Published	Thu, 8 Oct 2026 17:25:38 +0000

Description:

Given a dataset, filters out fields that have less than n percent of their rows populated.

Tags:

BigMLer BigMLer_ThuOct0826_170350

URL:

https://bigml.com/user/whizzml/gallery/script/6ac7b0d76a59afed275e482d

Script FREE

Source code

;; missing-count
;;
;; finds the number of missing datapoints for a given field in a dataset.
;;
;; Inputs:
;;   field-name: (string) name of field we want to inspect
;;   fields: (map) includes data for all fields
;;
;; Output: (integer) number of rows with missing data

(define (missing-count field-name fields)
  (fields [field-name "summary" "missing_count"] 0))

;; present-percent
;;
;; calculates the percentage of non-misisng data points for a field.
;;
;; Inputs:
;;   data: (map) includes data for all fields
;;   field-name: (string) name of field we want to inspect
;;   total-rows: (integer) the total number of rows in this dataset
;;
;; Output: (float) between 0 & 1

(define (present-percent data field-name total-rows)
  (let (fields (data "fields"))
    (- 1 (/ (missing-count field-name fields) total-rows))))

;; excluded-fields
;;
;; returns the field names for fields that will be excluded from the dataset
;; because the amount of missing data is greater than the specified threshold
;;
;; Inputs:
;;   dataset-id: (string) Dataset ID
;;   threshold: (float) Between 0 & 1. The precentange completeness of a row
;;                      we require
;;
;; Output: (list) List of field names that will be excluded

(define (excluded-fields dataset-id threshold)
  (let (data (fetch dataset-id)
        all-field-names (data "input_fields")
        total-rows (data "rows"))
    (filter (lambda (field-name)
              (> threshold (present-percent data field-name total-rows)))
            all-field-names)))

;; filtered-dataset
;;
;; Inputs:
;;   dataset-id: (string) Dataset ID - to be filtered
;;   threshold: (float) Between 0 & 1. The precentange completeness of a row
;;                      we require
;;
;; Output: (string) Dataset ID - filtered out rows based on threshold

(define (filtered-dataset dataset-id threshold)
  (let (excluded (excluded-fields dataset-id threshold)
        params (if (empty? excluded) {} {"excluded_fields" excluded}))
    (create-and-wait-dataset dataset-id params)))

(define output-dataset-id (filtered-dataset input-dataset-id threshold))

;; EXAMPLE:
;; (filtered-dataset "dataset/83hs0sj3819sdkd92k" 0.75) -> "dataset/kd8j83hf93h3hf83h99"

Description

Inputs

Outputs

License: Creative Commons Zero V1.0
Script FREE

Filter dataset on row population percentage | BigML.com

Sending Request...

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY

License

Embed this Script in your web site

COMPANY

PRODUCT

BUSINESS

TRAINING

GALLERY