- Notifications
You must be signed in to change notification settings - Fork62
Open
Description
Hivroom devs,
I encountered this head-scratching issue when runningvroom::problems() on a simple data.frame. In the original context I was running this in, a database connection was set up withDBI::dbConnect(), followed by a call toreadr::stop_for_problems() on CSV file that was being read in. I traced the error back tovroom::problems() and found that I could reproduce it with a trivial example:
library(DBI)con<- dbConnect(RSQLite::SQLite(),":memory:")input_file<-vroom::vroom_example("mtcars.csv")df<-vroom::vroom(input_file)#> Rows: 32 Columns: 12#> ── Column specification ────────────────────────────────────────────────────────#> Delimiter: ","#> chr (1): model#> dbl (11): mpg, cyl, disp, hp, drat, wt, qsec, vs, am, gear, carb#>#> ℹ Use `spec()` to retrieve the full column specification for this data.#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.vroom::problems(df)#> Error: vector memory limit of 16.0 Gb reached, see mem.maxVSize()
Session info
sessioninfo::session_info()#> ─ Session info ───────────────────────────────────────────────────────────────#> setting value#> version R version 4.4.2 (2024-10-31)#> os macOS Sequoia 15.3.1#> system aarch64, darwin20#> ui X11#> language en_US.UTF-8#> collate en_US.UTF-8#> ctype en_US.UTF-8#> tz Europe/London#> date 2025-03-10#> pandoc 3.6.3 @ /opt/homebrew/bin/ (via rmarkdown)#>#> ─ Packages ───────────────────────────────────────────────────────────────────#> package * version date (UTC) lib source#> bit 4.5.0.1 2024-12-03 [1] CRAN (R 4.4.1)#> bit64 4.6.0-1 2025-01-16 [1] CRAN (R 4.4.1)#> blob 1.2.4 2023-03-17 [1] CRAN (R 4.4.0)#> cachem 1.1.0 2024-05-16 [1] CRAN (R 4.4.0)#> cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0)#> crayon 1.5.3 2024-06-20 [1] CRAN (R 4.4.0)#> DBI * 1.2.3 2024-06-02 [1] CRAN (R 4.4.0)#> digest 0.6.37 2024-08-19 [1] CRAN (R 4.4.1)#> evaluate 1.0.1 2024-10-10 [1] CRAN (R 4.4.1)#> fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0)#> fs 1.6.5 2024-10-30 [1] CRAN (R 4.4.1)#> glue 1.8.0 2024-09-30 [1] CRAN (R 4.4.1)#> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)#> knitr 1.49 2024-11-08 [1] CRAN (R 4.4.1)#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0)#> memoise 2.0.1 2021-11-26 [1] CRAN (R 4.4.0)#> pillar 1.10.1 2025-01-07 [1] CRAN (R 4.4.1)#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)#> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.4.0)#> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.4.0)#> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.4.0)#> R.oo 1.27.0 2024-11-01 [1] CRAN (R 4.4.1)#> R.utils 2.12.3 2023-11-18 [1] CRAN (R 4.4.0)#> reprex 2.1.1 2024-07-06 [1] CRAN (R 4.4.0)#> rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0)#> rmarkdown 2.29 2024-11-04 [1] CRAN (R 4.4.1)#> RSQLite 2.3.9 2024-12-03 [1] CRAN (R 4.4.1)#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0)#> styler 1.10.3 2024-04-07 [1] CRAN (R 4.4.0)#> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.4.0)#> tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.4.0)#> tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.4.0)#> vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.4.0)#> vroom 1.6.5 2023-12-05 [1] CRAN (R 4.4.0)#> withr 3.0.2 2024-10-28 [1] CRAN (R 4.4.1)#> xfun 0.49 2024-10-31 [1] CRAN (R 4.4.1)#> yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0)#>#> [1] /Users/milan/Library/R/arm64/4.4/library#> [2] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library#>#> ──────────────────────────────────────────────────────────────────────────────
When I reverse the operations (i.e., runproblems() first and then set up the database connection) there is no problem at all.
By default, mymem.maxVSize() is set to16384 (16 GB) which should be more than enough?
Any idea what's going on here?
Metadata
Metadata
Assignees
Labels
No labels