Skip to Content

๋ฌธ์ œ ํ•ด๊ฒฐ

CoreRx ์šด์˜ ์ค‘ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ๋Š” ์ผ๋ฐ˜์ ์ธ ๋ฌธ์ œ์™€ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์„ ์ •๋ฆฌํ•ฉ๋‹ˆ๋‹ค. ์บ์‹œ ๋ถˆ์ผ์น˜, ETL ์‹คํŒจ, Docker/Superset ์ด์Šˆ๋ฅผ ๋‹ค๋ฃน๋‹ˆ๋‹ค.

์บ์‹œ ๊ด€๋ จ ๋ฌธ์ œ

๋Œ€์‹œ๋ณด๋“œ ๋ฐ์ดํ„ฐ๊ฐ€ ์˜ค๋ž˜๋จ

์ฆ์ƒ: ๋Œ€์‹œ๋ณด๋“œ์— ํ‘œ์‹œ๋˜๋Š” ๋ฐ์ดํ„ฐ๊ฐ€ ์ตœ์‹  ์—…๋กœ๋“œ ๋‚ด์šฉ์„ ๋ฐ˜์˜ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์›์ธ๊ณผ ํ•ด๊ฒฐ:

ํ™•์ธ ์ˆœ์„œํ™•์ธ ์‚ฌํ•ญํ•ด๊ฒฐ ๋ฐฉ๋ฒ•
1์บ์‹œ ๋ฉ”ํƒ€ ์ƒํƒœ ํ™•์ธ๋ฐ์ดํ„ฐ ๊ด€๋ฆฌ ํƒญ์—์„œ last_refreshed ํ™•์ธ
2์บ์‹œ ๊ฐฑ์‹ ์ด ์‹คํ–‰๋˜์ง€ ์•Š์Œrefresh_all_kpi_cache() ์ˆ˜๋™ ์‹คํ–‰
3์บ์‹œ ๊ฐฑ์‹ ์ด failed ์ƒํƒœerror_message ํ™•์ธ ํ›„ ์›์ธ ํ•ด๊ฒฐ
4ETL ์ž์ฒด๊ฐ€ ์‹คํ–‰๋˜์ง€ ์•Š์ŒETL ์Šคํฌ๋ฆฝํŠธ ์‹คํ–‰ ์ด๋ ฅ ํ™•์ธ

์บ์‹œ ๋ถˆ์ผ์น˜ (์›๋ณธ๊ณผ ์บ์‹œ ์ˆ˜์น˜ ๋‹ค๋ฆ„)

์ฆ์ƒ: SQL Lab์—์„œ ์›๋ณธ ํ…Œ์ด๋ธ”์„ ์ง์ ‘ ์กฐํšŒํ•œ ๊ฒฐ๊ณผ์™€ ์บ์‹œ ํ…Œ์ด๋ธ”์˜ ์ˆ˜์น˜๊ฐ€ ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

์›์ธ๊ณผ ํ•ด๊ฒฐ:

์›์ธํ•ด๊ฒฐ ๋ฐฉ๋ฒ•
์บ์‹œ ๊ฐฑ์‹  ํ›„ ์›๋ณธ ๋ฐ์ดํ„ฐ๊ฐ€ ๋ณ€๊ฒฝ๋จ์บ์‹œ ์žฌ๊ฐฑ์‹  ์‹คํ–‰
์บ์‹œ ๊ฐฑ์‹  ์ค‘ ์—๋Ÿฌ๋กœ ๋ถ€๋ถ„๋งŒ ๊ฐฑ์‹ ๋จ์ „์ฒด ์บ์‹œ ์žฌ๊ฐฑ์‹  ์‹คํ–‰
์›๋ณธ ์ฟผ๋ฆฌ์™€ ์บ์‹œ Function์˜ ์ง‘๊ณ„ ๊ธฐ์ค€ ์ฐจ์ด๊ธˆ์•ก ๋‹จ์œ„(์› vs ๋ฐฑ๋งŒ์›), ๊ธฐ๊ฐ„ ๋ฒ”์œ„, ํ•„ํ„ฐ ์กฐ๊ฑด ํ™•์ธ

์กฐ์น˜ ์ ˆ์ฐจ:

  1. corerx_cache_meta์—์„œ ํ•ด๋‹น ์บ์‹œ์˜ ์ƒํƒœ์™€ ๊ฐฑ์‹  ์‹œ๊ฐ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  2. ์ƒํƒœ๊ฐ€ failed์ด๋ฉด error_message๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  3. ์ „์ฒด ์บ์‹œ๋ฅผ ์žฌ๊ฐฑ์‹ ํ•ฉ๋‹ˆ๋‹ค.
SELECT refresh_all_kpi_cache( '4b5b1d04-23f6-42a7-9fe5-acb0a9998c55', 'manual' );

์บ์‹œ ๊ฐฑ์‹  ์‹คํŒจ (status = failed)

์ฆ์ƒ: corerx_cache_meta์˜ ์ƒํƒœ๊ฐ€ failed๋กœ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

์ผ๋ฐ˜์ ์ธ ์›์ธ:

์—๋Ÿฌ ๋ฉ”์‹œ์ง€์›์ธํ•ด๊ฒฐ
out of memoryPostgreSQL ๋ฉ”๋ชจ๋ฆฌ ๋ถ€์กฑwork_mem ์„ค์ • ํ™•์ธ ๋ฐ ์กฐ์ •
deadlock detected๋™์‹œ ๊ฐฑ์‹  ์ถฉ๋Œ๋‹จ์ผ ์„ธ์…˜์—์„œ ์žฌ์‹คํ–‰
relation does not existํ…Œ์ด๋ธ” ๋ฏธ์กด์žฌcreate_tables.py ์‹คํ–‰ ๋˜๋Š” ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ ํ™•์ธ
division by zero๋ถ„๋ชจ๊ฐ€ 0์ธ ๋ฐ์ดํ„ฐNULLIF ์ฒ˜๋ฆฌ ํ™•์ธ. DB Function ์ ๊ฒ€

ETL ๊ด€๋ จ ๋ฌธ์ œ

UBIST ์—‘์…€ ํŒŒ์‹ฑ ์‹คํŒจ

์ฆ์ƒ: ubist_import.py ์‹คํ–‰ ์‹œ ํ—ค๋” ํŒŒ์‹ฑ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์›์ธ๊ณผ ํ•ด๊ฒฐ:

์›์ธํ•ด๊ฒฐ ๋ฐฉ๋ฒ•
UBIST ์—‘์…€ ํŒŒ์ผ ํ˜•์‹ ๋ณ€๊ฒฝ์ƒˆ ํŒŒ์ผ์˜ ์ปฌ๋Ÿผ ๊ตฌ์กฐ ํ™•์ธ. ํ—ค๋” ํ–‰(1~2ํ–‰) ๋งคํ•‘ ์ฝ”๋“œ ์ˆ˜์ • ํ•„์š”
ํŒŒ์ผ์ด ์†์ƒ๋จUBIST์—์„œ ํŒŒ์ผ ์žฌ๋‹ค์šด๋กœ๋“œ
์ธ์ฝ”๋”ฉ ๋ฌธ์ œ์—‘์…€ ํŒŒ์ผ์„ ๋‹ค์‹œ ์ €์žฅ (UTF-8)

๋””๋ฒ„๊น… ์ ˆ์ฐจ:

  1. --dry-run ์˜ต์…˜์œผ๋กœ ๋จผ์ € ํ…Œ์ŠคํŠธํ•ฉ๋‹ˆ๋‹ค.
  2. ์—๋Ÿฌ ๋ฉ”์‹œ์ง€์—์„œ ์‹คํŒจํ•œ ํ–‰ ๋ฒˆํ˜ธ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  3. ํ•ด๋‹น ํ–‰์˜ ๋ฐ์ดํ„ฐ๋ฅผ ์—‘์…€์—์„œ ์ง์ ‘ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
python3 scripts/ubist_import.py --file "UBIST_D1 Sales.xlsx" --dry-run

ETL ์‹คํ–‰ ์ค‘ DB ์—ฐ๊ฒฐ ๋Š๊น€

์ฆ์ƒ: ETL ์‹คํ–‰ ์ค‘ connection reset ๋˜๋Š” server closed the connection ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์›์ธ๊ณผ ํ•ด๊ฒฐ:

์›์ธํ•ด๊ฒฐ ๋ฐฉ๋ฒ•
๋„คํŠธ์›Œํฌ ๋ถˆ์•ˆ์ •์„œ๋ฒ„ ๋„คํŠธ์›Œํฌ ์ƒํƒœ ํ™•์ธ ํ›„ ์žฌ์‹คํ–‰
PostgreSQL ํƒ€์ž„์•„์›ƒstatement_timeout ์„ค์ • ํ™•์ธ (๋Œ€์šฉ๋Ÿ‰ ETL์€ ์ œํ•œ ํ•ด์ œ ํ•„์š”)
Docker ์ปจํ…Œ์ด๋„ˆ ์žฌ์‹œ์ž‘Supabase Docker ์ƒํƒœ ํ™•์ธ (docker ps)

supply_qty ํƒ€์ž… ์—๋Ÿฌ

์ฆ์ƒ: ๋งค์ถœ ์ง‘๊ณ„ ์‹œ cannot cast text to numeric ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์›์ธ: kpis_gap_rows.supply_qty๋Š” TEXT ํƒ€์ž…์ž…๋‹ˆ๋‹ค. ํ•ฉ์‚ฐ ์‹œ ๋ฐ˜๋“œ์‹œ CAST๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ: ๋ชจ๋“  ์ฟผ๋ฆฌ์—์„œ supply_qty::numeric ๋˜๋Š” CAST(supply_qty AS numeric)์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

์ฃผ์˜: supply_qty ์ปฌ๋Ÿผ์€ ๊ธฐ์กด ํ…Œ์ด๋ธ”(kpis_gap_rows)์˜ ์ปฌ๋Ÿผ์ด๋ฏ€๋กœ ํƒ€์ž… ๋ณ€๊ฒฝ(ALTER)์ด ๊ธˆ์ง€๋ฉ๋‹ˆ๋‹ค. KPIS, CSO Web ์‚ฌ์ดํŠธ์—์„œ ๊ณต์œ ํ•˜๋Š” ํ…Œ์ด๋ธ”์ž…๋‹ˆ๋‹ค.

Docker/Superset ๊ด€๋ จ ๋ฌธ์ œ

Superset ์ปจํ…Œ์ด๋„ˆ ์‹œ์ž‘ ์‹คํŒจ

์ฆ์ƒ: Superset Docker ์ปจํ…Œ์ด๋„ˆ๊ฐ€ ์‹œ์ž‘๋˜์ง€ ์•Š๊ฑฐ๋‚˜, ์‹œ์ž‘ ํ›„ ์ฆ‰์‹œ ์ข…๋ฃŒ๋ฉ๋‹ˆ๋‹ค.

ํ™•์ธ ์ ˆ์ฐจ:

  1. Docker ์ปจํ…Œ์ด๋„ˆ ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
sudo docker compose -f docker-compose.superset.yml ps
  1. ์‹คํŒจํ•œ ์ปจํ…Œ์ด๋„ˆ์˜ ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
sudo docker compose -f docker-compose.superset.yml logs --tail 100

์ผ๋ฐ˜์ ์ธ ์›์ธ:

์›์ธํ•ด๊ฒฐ ๋ฐฉ๋ฒ•
ํฌํŠธ ์ถฉ๋Œ (8050)ํ•ด๋‹น ํฌํŠธ๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋‹ค๋ฅธ ํ”„๋กœ์„ธ์Šค ํ™•์ธ ๋ฐ ์ข…๋ฃŒ
๋ฉ”๋ชจ๋ฆฌ ๋ถ€์กฑ์„œ๋ฒ„ ๋ฉ”๋ชจ๋ฆฌ ํ™•์ธ. ๋ถˆํ•„์š”ํ•œ ์ปจํ…Œ์ด๋„ˆ ์ •๋ฆฌ
๋ณผ๋ฅจ ๊ถŒํ•œ ๋ฌธ์ œDocker ๋ณผ๋ฅจ ๊ถŒํ•œ ํ™•์ธ

Superset ๋Œ€์‹œ๋ณด๋“œ ์ ‘์† ๋ถˆ๊ฐ€

์ฆ์ƒ: ๋ธŒ๋ผ์šฐ์ €์—์„œ ๋Œ€์‹œ๋ณด๋“œ URL์— ์ ‘์†ํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.

ํ™•์ธ ์ ˆ์ฐจ:

ํ™•์ธ ์ˆœ์„œํ™•์ธ ์‚ฌํ•ญ๋ช…๋ น์–ด
1Docker ์ปจํ…Œ์ด๋„ˆ ์‹คํ–‰ ์ค‘sudo docker ps
2Superset ํฌํŠธ ๋ฆฌ์Šค๋‹sudo ss -tlnp | grep 8050
3Nginx ํ”„๋ก์‹œ ๋™์ž‘sudo nginx -t && sudo systemctl status nginx
4SSL ์ธ์ฆ์„œ ์œ ํšจsudo certbot certificates

Superset ์žฌ๋ฐฐํฌ

๋ฌธ์ œ ํ•ด๊ฒฐ ํ›„ Superset์„ ์žฌ๋ฐฐํฌํ•ด์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ:

bash scripts/redeploy_superset.sh

์ด ์Šคํฌ๋ฆฝํŠธ๋Š” Docker ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์ค‘์ง€, ์žฌ๋นŒ๋“œ, ์žฌ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค.

PostgreSQL ๊ด€๋ จ ๋ฌธ์ œ

DB ์—ฐ๊ฒฐ ์‹คํŒจ

์ฆ์ƒ: ETL ์Šคํฌ๋ฆฝํŠธ ๋˜๋Š” Superset์—์„œ DB ์—ฐ๊ฒฐ์ด ์‹คํŒจํ•ฉ๋‹ˆ๋‹ค.

ํ™•์ธ ์ ˆ์ฐจ:

  1. Supabase Docker ์ปจํ…Œ์ด๋„ˆ ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
sudo docker ps | grep supabase
  1. PostgreSQL ํฌํŠธ(5432)๊ฐ€ ๋ฆฌ์Šค๋‹ ์ค‘์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
sudo ss -tlnp | grep 5432
  1. ์ง์ ‘ ์—ฐ๊ฒฐ์„ ์‹œ๋„ํ•ฉ๋‹ˆ๋‹ค.
psql -h localhost -p 5432 -U postgres -d postgres

๋””์Šคํฌ ์šฉ๋Ÿ‰ ๋ถ€์กฑ

์ฆ์ƒ: ETL ์‹คํ–‰ ๋˜๋Š” ์บ์‹œ ๊ฐฑ์‹  ์‹œ no space left on device ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

ํ™•์ธ ๋ฐ ํ•ด๊ฒฐ:

  1. ๋””์Šคํฌ ์‚ฌ์šฉ๋Ÿ‰์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
df -h
  1. Docker ์ด๋ฏธ์ง€/๋ณผ๋ฅจ ์ •๋ฆฌ๋กœ ๊ณต๊ฐ„์„ ํ™•๋ณดํ•ฉ๋‹ˆ๋‹ค.
sudo docker system prune -f
  1. PostgreSQL WAL ๋กœ๊ทธ๊ฐ€ ๊ณผ๋„ํ•˜๊ฒŒ ์Œ“์ธ ๊ฒฝ์šฐ ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

๊ณตํ†ต ์ง„๋‹จ ์ ˆ์ฐจ

๋ฌธ์ œ ๋ฐœ์ƒ ์‹œ ๋‹ค์Œ ์ˆœ์„œ๋กœ ์ง„๋‹จํ•ฉ๋‹ˆ๋‹ค:

  1. ์ฆ์ƒ ํ™•์ธ: ๋Œ€์‹œ๋ณด๋“œ ํ™”๋ฉด, ์—๋Ÿฌ ๋ฉ”์‹œ์ง€, ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค.
  2. ์„œ๋น„์Šค ์ƒํƒœ ํ™•์ธ: Docker ์ปจํ…Œ์ด๋„ˆ, Nginx, PostgreSQL ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  3. ๋กœ๊ทธ ํ™•์ธ: Superset ๋กœ๊ทธ, PostgreSQL ๋กœ๊ทธ, ETL ์Šคํฌ๋ฆฝํŠธ ์ถœ๋ ฅ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  4. ์›์ธ ํŠน์ •: ๋„คํŠธ์›Œํฌ, ๋””์Šคํฌ, ๋ฉ”๋ชจ๋ฆฌ, ๋ฐ์ดํ„ฐ ๋ฌธ์ œ ์ค‘ ์–ด๋–ค ๊ฒƒ์ธ์ง€ ๊ตฌ๋ถ„ํ•ฉ๋‹ˆ๋‹ค.
  5. ํ•ด๊ฒฐ ํ›„ ๊ฒ€์ฆ: ๋ฌธ์ œ ํ•ด๊ฒฐ ํ›„ ์บ์‹œ ๊ฐฑ์‹  โ†’ ๋Œ€์‹œ๋ณด๋“œ ํ™•์ธ์œผ๋กœ ์ •์ƒ ๋™์ž‘์„ ๊ฒ€์ฆํ•ฉ๋‹ˆ๋‹ค.

๋‹ค์Œ ๋‹จ๊ณ„

Last updated on